--- /dev/null
+\documentclass{article}
+
+\usepackage{imakeidx}
+\usepackage[pdfborder={0 0 0}]{hyperref}
+\usepackage{longtable}
+
+\title{bcachefs: Principles of Operation}
+\author{Kent Overstreet}
+
+\date{}
+
+\begin{document}
+
+\maketitle
+\tableofcontents
+
+\section{Introduction and overview}
+
+Bcachefs is a modern, general purpose, copy on write filesystem descended from
+bcache, a block layer cache.
+
+The internal architecture is very different from most existing filesystems where
+the inode is central and many data structures hang off of the inode. Instead,
+bcachefs is architected more like a filesystem on top of a relational database,
+with tables for the different filesystem data types - extents, inodes, dirents,
+xattrs, et cetera.
+
+bcachefs supports almost all of the same features as other modern COW
+filesystems, such as ZFS and btrfs, but in general with a cleaner, simpler,
+higher performance design.
+
+\subsection{Performance overview}
+
+The core of the architecture is a very high performance and very low latency b+
+tree, which also is not a conventional b+ tree but more of hybrid, taking
+concepts from compacting data structures: btree nodes are very large, log
+structured, and compacted (resorted) as necessary in memory. This means our b+
+trees are very shallow compared to other filesystems.
+
+What this means for the end user is that since we require very few seeks or disk
+reads, filesystem latency is extremely good - especially cache cold filesystem
+latency, which does not show up in most benchmarks but has a huge impact on real
+world performance, as well as how fast the system "feels" in normal interactive
+usage. Latency has been a major focus throughout the codebase - notably, we have
+assertions that we never hold b+ tree locks while doing IO, and the btree
+transaction layer makes it easily to aggressively drop and retake locks as
+needed - one major goal of bcachefs is to be the first general purpose soft
+realtime filesystem.
+
+Additionally, unlike other COW btrees, btree updates are journalled. This
+greatly improves our write efficiency on random update workloads, as it means
+btree writes are only done when we have a large block of updates, or when
+required by memory reclaim or journal reclaim.
+
+\subsection{Bucket based allocation}
+
+As mentioned bcachefs is descended from bcache, where the ability to efficiently
+invalidate cached data and reuse disk space was a core design requirement. To
+make this possible the allocator divides the disk up into buckets, typically
+512k to 2M but possibly larger or smaller. Buckets and data pointers have
+generation numbers: we can reuse a bucket with cached data in it without finding
+and deleting all the data pointers by incrementing the generation number.
+
+In keeping with the copy-on-write theme of avoiding update in place wherever
+possible, we never rewrite or overwrite data within a bucket - when we allocate
+a bucket, we write to it sequentially and then we don't write to it again until
+the bucket has been invalidated and the generation number incremented.
+
+This means we require a copying garbage collector to deal with internal
+fragmentation, when patterns of random writes leave us with many buckets that
+are partially empty (because the data they contained was overwritten) - copy GC
+evacuates buckets that are mostly empty by writing the data they contain to new
+buckets. This also means that we need to reserve space on the device for the
+copy GC reserve when formatting - typically 8\% or 12\%.
+
+There are some advantages to structuring the allocator this way, besides being
+able to support cached data:
+\begin{itemize}
+ \item By maintaining multiple write points that are writing to different buckets,
+ we're able to easily and naturally segregate unrelated IO from different
+ processes, which helps greatly with fragmentation.
+
+ \item The fast path of the allocator is essentially a simple bump allocator - the
+ disk space allocation is extremely fast
+
+ \item Fragmentation is generally a non issue unless copygc has to kick
+ in, and it usually doesn't under typical usage patterns. The
+ allocator and copygc are doing essentially the same things as
+ the flash translation layer in SSDs, but within the filesystem
+ we have much greater visibility into where writes are coming
+ from and how to segregate them, as well as which data is
+ actually live - performance is generally more predictable than
+ with SSDs under similar usage patterns.
+
+ \item The same algorithms will in the future be used for managing SMR
+ hard drives directly, avoiding the translation layer in the hard
+ drive - doing this work within the filesystem should give much
+ better performance and much more predictable latency.
+\end{itemize}
+
+\section{Feature overview}
+
+\subsection{IO path options}
+
+Most options that control the IO path can be set at either the filesystem level
+or on individual inodes (files and directories). When set on a directory via the
+\texttt{bcachefs attr} command, they will be automatically applied recursively.
+
+\subsubsection{Checksumming}
+
+bcachefs supports both metadata and data checksumming - crc32c by default, but
+stronger checksums are available as well. Enabling data checksumming incurs some
+performance overhead - besides the checksum calculation, writes have to be
+bounced for checksum stability (Linux generally cannot guarantee that the buffer
+being written is not modified in flight), but reads generally do not have to be
+bounced.
+
+Checksum granularity in bcachefs is at the level of individual extents, which
+results in smaller metadata but means we have to read entire extents in order to
+verify the checksum. By default, checksummed and compressed extents are capped
+at 64k. For most applications and usage scenarios this is an ideal trade off, but
+small random \texttt{O\_DIRECT} reads will incur significant overhead. In the
+future, checksum granularity will be a per-inode option.
+
+\subsubsection{Encryption}
+
+bcachefs supports authenticated (AEAD style) encryption - ChaCha20/Poly1305.
+When encryption is enabled, the poly1305 MAC replaces the normal data and
+metadata checksums. This style of encryption is superior to typical block layer
+or filesystem level encryption (usually AES-XTS), which only operates on blocks
+and doesn't have a way to store nonces or MACs. In contrast, we store a nonce
+and cryptographic MAC alongside data pointers - meaning we have a chain of trust
+up to the superblock (or journal, in the case of unclean shutdowns) and can
+definitely tell if metadata has been modified, dropped, or replaced with an
+earlier version - replay attacks are not possible.
+
+Encryption can only be specified for the entire filesystem, not per file or
+directory - this is because metadata blocks do not belong to a particular file.
+All metadata except for the superblock is encrypted.
+
+In the future we'll probably add AES-GCM for platforms that have hardware
+acceleration for AES, but in the meantime software implementations of ChaCha20
+are also quite fast on most platforms.
+
+\texttt{scrypt} is used for the key derivation function - for converting the
+user supplied passphrase to an encryption key.
+
+To format a filesystem with encryption, use
+\begin{quote} \begin{verbatim}
+bcachefs format --encrypted /dev/sda1
+\end{verbatim} \end{quote}
+
+You will be prompted for a passphrase. Then, to use an encrypted filesystem
+use the command
+\begin{quote} \begin{verbatim}
+bcachefs unlock /dev/sda1
+\end{verbatim} \end{quote}
+
+You will be prompted for the passphrase and the encryption key will be added to
+your in-kernel keyring; mount, fsck and other commands will then work as usual.
+
+The passphrase on an existing encrypted filesystem can be changed with the
+\texttt{bcachefs set-passphrase} command. To permanently unlock an encrypted
+filesystem, use the \texttt{bcachefs remove-passphrase} command - this can be
+useful when dumping filesystem metadata for debugging by the developers.
+
+There is a \texttt{wide\_macs} option which controls the size of the
+cryptographic MACs stored on disk. By default, only 80 bits are stored, which
+should be sufficient security for most applications. With the
+\texttt{wide\_macs} option enabled we store the full 128 bit MAC, at the cost of
+making extents 8 bytes bigger.
+
+\subsubsection{Compression}
+
+bcachefs supports gzip, lz4 and zstd compression. As with data checksumming, we
+compress entire extents, not individual disk blocks - this gives us better
+compression ratios than other filesystems, at the cost of reduced small random
+read performance.
+
+Data can also be compressed or recompressed with a different algorithm in the
+background by the rebalance thread, if the \texttt{background\_compression}
+option is set.
+
+\subsection{Multiple devices}
+
+bcachefs is a multi-device filesystem. Devices need not be the same size: by
+default, the allocator will stripe across all available devices but biasing in
+favor of the devices with more free space, so that all devices in the filesystem
+fill up at the same rate. Devices need not have the same performance
+characteristics: we track device IO latency and direct reads to the device that
+is currently fastest.
+
+\subsubsection{Replication}
+
+bcachefs supports standard RAID1/10 style redundancy with the
+\texttt{data\_replicas} and \texttt{metadata\_replicas} options. Layout is not
+fixed as with RAID10: a given extent can be replicated across any set of
+devices; the \texttt{bcachefs fs usage} command shows how data is replicated
+within a filesystem.
+
+\subsubsection{Erasure coding}
+
+bcachefs also supports Reed-Solomon erasure coding - the same algorithm used by
+most RAID5/6 implementations) When enabled with the \texttt{ec} option, the
+desired redundancy is taken from the \texttt{data\_replicas} option - erasure
+coding of metadata is not supported.
+
+Erasure coding works significantly differently from both conventional RAID
+implementations and other filesystems with similar features. In conventional
+RAID, the "write hole" is a significant problem - doing a small write within a
+stripe requires the P and Q (recovery) blocks to be updated as well, and since
+those writes cannot be done atomically there is a window where the P and Q
+blocks are inconsistent - meaning that if the system crashes and recovers with a
+drive missing, reconstruct reads for unrelated data within that stripe will be
+corrupted.
+
+ZFS avoids this by fragmenting individual writes so that every write becomes a
+new stripe - this works, but the fragmentation has a negative effect on
+performance: metadata becomes bigger, and both read and write requests are
+excessively fragmented. Btrfs's erasure coding implementation is more
+conventional, and still subject to the write hole problem.
+
+bcachefs's erasure coding takes advantage of our copy on write nature - since
+updating stripes in place is a problem, we simply don't do that. And since
+excessively small stripes is a problem for fragmentation, we don't erasure code
+individual extents, we erasure code entire buckets - taking advantage of bucket
+based allocation and copying garbage collection.
+
+When erasure coding is enabled, writes are initially replicated, but one of the
+replicas is allocated from a bucket that is queued up to be part of a new
+stripe. When we finish filling up the new stripe, we write out the P and Q
+buckets and then drop the extra replicas for all the data within that stripe -
+the effect is similar to full data journalling, and it means that after erasure
+coding is done the layout of our data on disk is ideal.
+
+Since disks have write caches that are only flushed when we issue a cache flush
+command - which we only do on journal commit - if we can tweak the allocator so
+that the buckets used for the extra replicas are reused (and then overwritten
+again) immediately, this full data journalling should have negligible overhead -
+this optimization is not implemented yet, however.
+
+\subsubsection{Device labels and targets}
+
+By default, writes are striped across all devices in a filesystem, but they may
+be directed to a specific device or set of devices with the various target
+options. The allocator only prefers to allocate from devices matching the
+specified target; if those devices are full, it will fall back to allocating
+from any device in the filesystem.
+
+Target options may refer to a device directly, e.g.
+\texttt{foreground\_target=/dev/sda1}, or they may refer to a device label. A
+device label is a path delimited by periods - e.g. ssd.ssd1 (and labels need not
+be unique). This gives us ways of referring to multiple devices in target
+options: If we specify ssd in a target option, that will refer to all devices
+with the label ssd or labels that start with ssd. (e.g. ssd.ssd1, ssd.ssd2).
+
+Four target options exist. These options all may be set at the filesystem level
+(at format time, at mount time, or at runtime via sysfs), or on a particular
+file or directory:
+
+\begin{description}
+ \item \texttt{foreground\_target}: normal foreground data writes, and
+ metadata if \\ \texttt{metadata\_target} is not set
+ \item \texttt{metadata\_target}: btree writes
+ \item \texttt{background\_target}: If set, user data (not metadata) will
+ be moved to this target in the background
+ \item\texttt{promote\_target}: If set, a cached copy will be added to
+ this target on read, if none exists
+\end{description}
+
+\subsubsection{Caching}
+
+When an extent has multiple copies on different devices, some of those copies
+may be marked as cached. Buckets containing only cached data are discarded as
+needed by the allocator in LRU order.
+
+When data is moved from one device to another according to the \\
+\texttt{background\_target} option, the original copy is left in place but
+marked as cached. With the \texttt{promote\_target} option, the original copy is
+left unchanged and the new copy on the \texttt{promote\_target} device is marked
+as cached.
+
+To do writeback caching, set \texttt{foreground\_target} and
+\texttt{promote\_target} to the cache device, and \texttt{background\_target} to
+the backing device. To do writearound caching, set \texttt{foreground\_target}
+to the backing device and \texttt{promote\_target} to the cache device.
+
+\subsubsection{Durability}
+
+Some devices may be considered to be more reliable than others. For example, we
+might have a filesystem composed of a hardware RAID array and several NVME flash
+devices, to be used as cache. We can set replicas=2 so that losing any of the
+NVME flash devices will not cause us to lose data, and then additionally we can
+set durability=2 for the hardware RAID device to tell bcachefs that we don't
+need extra replicas for data on that device - data on that device will count as
+two replicas, not just one.
+
+The durability option can also be used for writethrough caching: by setting
+durability=0 for a device, it can be used as a cache and only as a cache -
+bcachefs won't consider copies on that device to count towards the number of
+replicas we're supposed to keep.
+
+\subsection{Reflink}
+
+bcachefs supports reflink, similarly to other filesystems with the same feature.
+cp --reflink will create a copy that shares the underlying storage. Reading from
+that file will become slightly slower - the extent pointing to that data is
+moved to the reflink btree (with a refcount added) and in the extents btree we
+leave a key that points to the indirect extent in the reflink btree, meaning
+that we now have to do two btree lookups to read from that data instead of just
+one.
+
+\subsection{Inline data extents}
+
+bcachefs supports inline data extents, controlled by the \texttt{inline\_data}
+option (on by default). When the end of a file is being written and is smaller
+than half of the filesystem blocksize, it will be written as an inline data
+extent. Inline data extents can also be reflinked (moved to the reflink btree
+with a refcount added): as a todo item we also intend to support compressed
+inline data extents.
+
+\subsection{Subvolumes and snapshots}
+
+bcachefs supports subvolumes and snapshots with a similar userspace interface as
+btrfs. A new subvolume may be created empty, or it may be created as a snapshot
+of another subvolume. Snapshots are writeable and may be snapshotted again,
+creating a tree of snapshots.
+
+Snapshots are very cheap to create: they're not based on cloning of COW btrees
+as with btrfs, but instead are based on versioning of individual keys in the
+btrees. Many thousands or millions of snapshots can be created, with the only
+limitation being disk space.
+
+The following subcommands exist for managing subvolumes and snapshots:
+\begin{itemize}
+ \item \texttt{bcachefs subvolume create}: Create a new, empty subvolume
+ \item \texttt{bcachefs subvolume destroy}: Delete an existing subvolume
+ or snapshot
+ \item \texttt{bcachefs subvolume snapshot}: Create a snapshot of an
+ existing subvolume
+\end{itemize}
+
+A subvolume can also be deleting with a normal rmdir after deleting all the
+contents, as with \texttt{rm -rf}. Still to be implemented: read-only snapshots,
+recursive snapshot creation, and a method for recursively listing subvolumes.
+
+\subsection{Quotas}
+
+bcachefs supports conventional user/group/project quotas. Quotas do not
+currently apply to snapshot subvolumes, because if a file changes ownership in
+the snapshot it would be ambiguous as to what quota data within that file
+should be charged to.
+
+When a directory has a project ID set it is inherited automatically by
+descendants on creation and rename. When renaming a directory would cause the
+project ID to change we return -EXDEV so that the move is done file by file, so
+that the project ID is propagated correctly to descendants - thus, project
+quotas can be used as subdirectory quotas.
+
+\section{Management}
+
+\subsection{Formatting}
+
+To format a new bcachefs filesystem use the subcommand \texttt{bcachefs
+format}, or \texttt{mkfs.bcachefs}. All persistent filesystem-wide options can
+be specified at format time. For an example of a multi device filesystem with
+compression, encryption, replication and writeback caching:
+\begin{quote} \begin{verbatim}
+bcachefs format --compression=lz4 \
+ --encrypted \
+ --replicas=2 \
+ --label=ssd.ssd1 /dev/sda \
+ --label=ssd.ssd2 /dev/sdb \
+ --label=hdd.hdd1 /dev/sdc \
+ --label=hdd.hdd2 /dev/sdd \
+ --label=hdd.hdd3 /dev/sde \
+ --label=hdd.hdd4 /dev/sdf \
+ --foreground_target=ssd \
+ --promote_target=ssd \
+ --background_target=hdd
+\end{verbatim} \end{quote}
+
+\subsection{Mounting}
+
+To mount a multi device filesystem, there are two options. You can specify all
+component devices, separated by hyphens, e.g.
+\begin{quote} \begin{verbatim}
+mount -t bcachefs /dev/sda:/dev/sdb:/dev/sdc /mnt
+\end{verbatim} \end{quote}
+Or, use the mount.bcachefs tool to mount by filesystem UUID. Still todo: improve
+the mount.bcachefs tool to support mounting by filesystem label.
+
+No special handling is needed for recovering from unclean shutdown. Journal
+replay happens automatically, and diagnostic messages in the dmesg log will
+indicate whether recovery was from clean or unclean shutdown.
+
+The \texttt{-o degraded} option will allow a filesystem to be mounted without
+all the the devices, but will fail if data would be missing. The
+\texttt{-o very\_degraded} can be used to attempt mounting when data would be
+missing.
+
+Also relevant is the \texttt{-o nochanges} option. It disallows any and all
+writes to the underlying devices, pinning dirty data in memory as necessary if
+for example journal replay was necessary - think of it as a "super read-only"
+mode. It can be used for data recovery, and for testing version upgrades.
+
+The \texttt{-o verbose} enables additional log output during the mount process.
+
+\subsection{Fsck}
+
+It is possible to run fsck either in userspace with the \texttt{bcachefs fsck}
+subcommand (also available as \texttt{fsck.bcachefs}, or in the kernel while
+mounting by specifying the \texttt{-o fsck} mount option. In either case the
+exact same fsck implementation is being run, only the environment is different.
+Running fsck in the kernel at mount time has the advantage of somewhat better
+performance, while running in userspace has the ability to be stopped with
+ctrl-c and can prompt the user for fixing errors. To fix errors while running
+fsck in the kernel, use the \texttt{-o fix\_errors} option.
+
+The \texttt{-n} option passed to fsck implies the \texttt{-o nochanges} option;
+\texttt{bcachefs fsck -ny} can be used to test filesystem repair in dry-run
+mode.
+
+\subsection{Status of data}
+
+The \texttt{bcachefs fs usage} may be used to display filesystem usage broken
+out in various ways. Data usage is broken out by type: superblock, journal,
+btree, data, cached data, and parity, and by which sets of devices extents are
+replicated across. We also give per-device usage which includes fragmentation
+due to partially used buckets.
+
+\subsection{Journal}
+
+The journal has a number of tunables that affect filesystem performance. Journal
+commits are fairly expensive operations as they require issuing FLUSH and FUA
+operations to the underlying devices. By default, we issue a journal flush one
+second after a filesystem update has been done; this is controlled with the
+\texttt{journal\_flush\_delay} option, which takes a parameter in milliseconds.
+
+Filesystem sync and fsync operations issue journal flushes; this can be disabled
+with the \texttt{journal\_flush\_disabled} option - the
+\texttt{journal\_flush\_delay} option will still apply, and in the event of a
+system crash we will never lose more than (by default) one second of work. This
+option may be useful on a personal workstation or laptop, and perhaps less
+appropriate on a server.
+
+The journal reclaim thread runs in the background, kicking off btree node writes
+and btree key cache flushes to free up space in the journal. Even in the absence
+of space pressure it will run slowly in the background: this is controlled by
+the \texttt{journal\_reclaim\_delay} parameter, with a default of 100
+milliseconds.
+
+The journal should be sized sufficiently that bursts of activity do not fill up
+the journal too quickly; also, a larger journal mean that we can queue up larger
+btree writes. The \texttt{bcachefs device resize-journal} can be used for
+resizing the journal on disk on a particular device - it can be used on a
+mounted or unmounted filesystem.
+
+In the future, we should implement a method to see how much space is currently
+utilized in the journal.
+
+\subsection{Device management}
+
+\subsubsection{Filesystem resize}
+
+A filesystem can be resized on a particular device with the
+\texttt{bcachefs device resize} subcommand. Currently only growing is supported,
+not shrinking.
+
+\subsubsection{Device add/removal}
+
+The following subcommands exist for adding and removing devices from a mounted
+filesystem:
+\begin{itemize}
+ \item \texttt{bcachefs device add}: Formats and adds a new device to an
+ existing filesystem.
+ \item \texttt{bcachefs device remove}: Permenantly removes a device from
+ an existing filesystem.
+ \item \texttt{bcachefs device online}: Connects a device to a running
+ filesystem that was mounted without it (i.e. in degraded mode)
+ \item \texttt{bcachefs device offline}: Disconnects a device from a
+ mounted filesystem without removing it.
+ \item \texttt{bcachefs device evacuate}: Migrates data off of a
+ particular device to prepare for removal, setting it read-only
+ if necessary.
+ \item \texttt{bcachefs device set-state}: Changes the state of a member
+ device: one of rw (readwrite), ro (readonly), failed, or spare.
+
+ A failed device is considered to have 0 durability, and replicas
+ on that device won't be counted towards the number of replicas
+ an extent should have by rereplicate - however, bcachefs will
+ still attempt to read from devices marked as failed.
+\end{itemize}
+
+The \texttt{bcachefs device remove}, \texttt{bcachefs device offline} and
+\texttt{bcachefs device set-state} commands take force options for when they
+would leave the filesystem degraded or with data missing. Todo: regularize and
+improve those options.
+
+\subsection{Data management}
+
+\subsubsection{Data rereplicate}
+
+The \texttt{bcachefs data rereplicate} command may be used to scan for extents
+that have insufficient replicas and write additional replicas, e.g. after a
+device has been removed from a filesystem or after replication has been enabled
+or increased.
+
+\subsubsection{Rebalance}
+
+To be implemented: a command for moving data between devices to equalize usage
+on each device. Not normally required because the allocator attempts to equalize
+usage across devices as it stripes, but can be necessary in certain scenarios -
+i.e. when a two-device filesystem with replication enabled that is very full has
+a third device added.
+
+\subsubsection{Scrub}
+
+To be implemented: a command for reading all data within a filesystem and
+ensuring that checksums are valid, fixing bitrot when a valid copy can be found.
+
+\section{Options}
+
+Most bcachefs options can be set filesystem wide, and a significant subset can
+also be set on inodes (files and directories), overriding the global defaults.
+Filesystem wide options may be set when formatting, when mounting, or at runtime
+via \texttt{/sys/fs/bcachefs/<uuid>/options/}. When set at runtime via sysfs the
+persistent options in the superblock are updated as well; when options are
+passed as mount parameters the persistent options are unmodified.
+
+\subsection{File and directory options}
+
+Options set on inodes (files and directories) are automatically inherited by
+their descendants, and inodes also record whether a given option was explicitly
+set or inherited from their parent. When renaming a directory would cause
+inherited attributes to change we fail the rename with -EXDEV, causing userspace
+to do the rename file by file so that inherited attributes stay consistent.
+
+Inode options are available as extended attributes. The options that have been
+explicitly set are available under the \texttt{bcachefs} namespace, and the effective
+options (explicitly set and inherited options) are available under the
+\texttt{bcachefs\_effective} namespace. Examples of listing options with the
+getfattr command:
+
+\begin{quote} \begin{verbatim}
+$ getfattr -d -m '^bcachefs\.' filename
+$ getfattr -d -m '^bcachefs_effective\.' filename
+\end{verbatim} \end{quote}
+
+Options may be set via the extended attribute interface, but it is preferable to
+use the \texttt{bcachefs setattr} command as it will correctly propagate options
+recursively.
+
+\subsection{Full option list}
+
+\begin{tabbing}
+\hspace{0.2in} \= \kill
+ \texttt{block\_size} \` \textbf{format} \\
+ \> \parbox{4.3in}{Filesystem block size (default 4k)} \\ \\
+
+ \texttt{btree\_node\_size} \` \textbf{format} \\
+ \> Btree node size, default 256k \\ \\
+
+ \texttt{errors} \` \textbf{format,mount,rutime} \\
+ \> Action to take on filesystem error \\ \\
+
+ \texttt{metadata\_replicas} \` \textbf{format,mount,runtime} \\
+ \> Number of replicas for metadata (journal and btree) \\ \\
+
+ \texttt{data\_replicas} \` \textbf{format,mount,runtime,inode} \\
+ \> Number of replicas for user data \\ \\
+
+ \texttt{replicas} \` \textbf{format} \\
+ \> Alias for both metadata\_replicas and data\_replicas \\ \\
+
+ \texttt{metadata\_checksum} \` \textbf{format,mount,runtime} \\
+ \> Checksum type for metadata writes \\ \\
+
+ \texttt{data\_checksum} \` \textbf{format,mount,runtime,inode} \\
+ \> Checksum type for data writes \\ \\
+
+ \texttt{compression} \` \textbf{format,mount,runtime,inode} \\
+ \> Compression type \\ \\
+
+ \texttt{background\_compression} \` \textbf{format,mount,runtime,inode} \\
+ \> Background compression type \\ \\
+
+ \texttt{str\_hash} \` \textbf{format,mount,runtime,inode} \\
+ \> Hash function for string hash tables (directories and xattrs) \\ \\
+
+ \texttt{metadata\_target} \` \textbf{format,mount,runtime,inode} \\
+ \> Preferred target for metadata writes \\ \\
+
+ \texttt{foreground\_target} \` \textbf{format,mount,runtime,inode} \\
+ \> Preferred target for foreground writes \\ \\
+
+ \texttt{background\_target} \` \textbf{format,mount,runtime,inode} \\
+ \> Target for data to be moved to in the background \\ \\
+
+ \texttt{promote\_target} \` \textbf{format,mount,runtime,inode} \\
+ \> Target for data to be copied to on read \\ \\
+
+ \texttt{erasure\_code} \` \textbf{format,mount,runtime,inode} \\
+ \> Enable erasure coding \\ \\
+
+ \texttt{inodes\_32bit} \` \textbf{format,mount,runtime} \\
+ \> Restrict new inode numbers to 32 bits \\ \\
+
+ \texttt{shard\_inode\_numbers} \` \textbf{format,mount,runtime} \\
+ \> Use CPU id for high bits of new inode numbers. \\ \\
+
+ \texttt{wide\_macs} \` \textbf{format,mount,runtime} \\
+ \> Store full 128 bit cryptographic MACs (default 80) \\ \\
+
+ \texttt{inline\_data} \` \textbf{format,mount,runtime} \\
+ \> Enable inline data extents (default on) \\ \\
+
+ \texttt{journal\_flush\_delay} \` \textbf{format,mount,runtime} \\
+ \> Delay in milliseconds before automatic journal commit (default 1000) \\ \\
+
+ \texttt{journal\_flush\_disabled}\`\textbf{format,mount,runtime} \\
+ \> \begin{minipage}{4.3in}Disables journal flush on sync/fsync.
+ \texttt{journal\_flush\_delay} remains in effect, thus with the
+ default setting not more than 1 second of work will be lost.
+ \end{minipage} \\ \\
+
+ \texttt{journal\_reclaim\_delay}\` \textbf{format,mount,runtime} \\
+ \> Delay in milliseconds before automatic journal reclaim \\ \\
+
+ \texttt{acl} \` \textbf{format,mount} \\
+ \> Enable POSIX ACLs \\ \\
+
+ \texttt{usrquota} \` \textbf{format,mount} \\
+ \> Enable user quotas \\ \\
+
+ \texttt{grpquota} \` \textbf{format,mount} \\
+ \> Enable group quotas \\ \\
+
+ \texttt{prjquota} \` \textbf{format,mount} \\
+ \> Enable project quotas \\ \\
+
+ \texttt{degraded} \` \textbf{mount} \\
+ \> Allow mounting with data degraded \\ \\
+
+ \texttt{very\_degraded} \` \textbf{mount} \\
+ \> Allow mounting with data missing \\ \\
+
+ \texttt{verbose} \` \textbf{mount} \\
+ \> Extra debugging info during mount/recovery \\ \\
+
+ \texttt{fsck} \` \textbf{mount} \\
+ \> Run fsck during mount \\ \\
+
+ \texttt{fix\_errors} \` \textbf{mount} \\
+ \> Fix errors without asking during fsck \\ \\
+
+ \texttt{ratelimit\_errors} \` \textbf{mount} \\
+ \> Ratelimit error messages during fsck \\ \\
+
+ \texttt{read\_only} \` \textbf{mount} \\
+ \> Mount in read only mode \\ \\
+
+ \texttt{nochanges} \` \textbf{mount} \\
+ \> Issue no writes, even for journal replay \\ \\
+
+ \texttt{norecovery} \` \textbf{mount} \\
+ \> Don't replay the journal (not recommended) \\ \\
+
+ \texttt{noexcl} \` \textbf{mount} \\
+ \> Don't open devices in exclusive mode \\ \\
+
+ \texttt{version\_upgrade} \` \textbf{mount} \\
+ \> Upgrade on disk format to latest version \\ \\
+
+ \texttt{discard} \` \textbf{device} \\
+ \> Enable discard/TRIM support \\ \\
+\end{tabbing}
+
+\subsection{Error actions}
+The \texttt{errors} option is used for inconsistencies that indicate some sort
+of a bug. Valid error actions are:
+\begin{description}
+ \item[{\tt continue}] Log the error but continue normal operation
+ \item[{\tt ro}] Emergency read only, immediately halting any changes
+ to the filesystem on disk
+ \item[{\tt panic}] Immediately halt the entire machine, printing a
+ backtrace on the system console
+\end{description}
+
+\subsection{Checksum types}
+Valid checksum types are:
+\begin{description}
+ \item[{\tt none}]
+ \item[{\tt crc32c}] (default)
+ \item[{\tt crc64}]
+\end{description}
+
+\subsection{Compression types}
+Valid compression types are:
+\begin{description}
+ \item[{\tt none}] (default)
+ \item[{\tt lz4}]
+ \item[{\tt gzip}]
+ \item[{\tt zstd}]
+\end{description}
+
+\subsection{String hash types}
+Valid hash types for string hash tables are:
+\begin{description}
+ \item[{\tt crc32c}]
+ \item[{\tt crc64}]
+ \item[{\tt siphash}] (default)
+\end{description}
+
+\section{Debugging tools}
+
+\subsection{Sysfs interface}
+
+Mounted filesystems are available in sysfs at \texttt{/sys/fs/bcachefs/<uuid>/}
+with various options, performance counters and internal debugging aids.
+
+\subsubsection{Options}
+
+Filesystem options may be viewed and changed via \\
+\texttt{/sys/fs/bcachefs/<uuid>/options/}, and settings changed via sysfs will
+be persistently changed in the superblock as well.
+
+\subsubsection{Time stats}
+
+bcachefs tracks the latency and frequency of various operations and events, with
+quantiles for latency/duration in the
+\texttt{/sys/fs/bcachefs/<uuid>/time\_stats/} directory.
+
+\begin{description}
+ \item \texttt{blocked\_allocate} \\
+ Tracks when allocating a bucket must wait because none are
+ immediately available, meaning the copygc thread is not keeping
+ up with evacuating mostly empty buckets or the allocator thread
+ is not keeping up with invalidating and discarding buckets.
+
+ \item \texttt{blocked\_allocate\_open\_bucket} \\
+ Tracks when allocating a bucket must wait because all of our
+ handles for pinning open buckets are in use (we statically
+ allocate 1024).
+
+ \item \texttt{blocked\_journal} \\
+ Tracks when getting a journal reservation must wait, either
+ because journal reclaim isn't keeping up with reclaiming space
+ in the journal, or because journal writes are taking too long to
+ complete and we already have too many in flight.
+
+ \item \texttt{btree\_gc} \\
+ Tracks when the btree\_gc code must walk the btree at runtime -
+ for recalculating the oldest outstanding generation number of
+ every bucket in the btree.
+
+ \item \texttt{btree\_lock\_contended\_read}
+ \item \texttt{btree\_lock\_contended\_intent}
+ \item \texttt{btree\_lock\_contended\_write} \\
+ Track when taking a read, intent or write lock on a btree node
+ must block.
+
+ \item \texttt{btree\_node\_mem\_alloc} \\
+ Tracks the total time to allocate memory in the btree node cache
+ for a new btree node.
+
+ \item \texttt{btree\_node\_split} \\
+ Tracks btree node splits - when a btree node becomes full and is
+ split into two new nodes
+
+ \item \texttt{btree\_node\_compact} \\
+ Tracks btree node compactions - when a btree node becomes full
+ and needs to be compacted on disk.
+
+ \item \texttt{btree\_node\_merge} \\
+ Tracks when two adjacent btree nodes are merged.
+
+ \item \texttt{btree\_node\_sort} \\
+ Tracks sorting and resorting entire btree nodes in memory,
+ either after reading them in from disk or for compacting prior
+ to creating a new sorted array of keys.
+
+ \item \texttt{btree\_node\_read} \\
+ Tracks reading in btree nodes from disk.
+
+ \item \texttt{btree\_interior\_update\_foreground} \\
+ Tracks foreground time for btree updates that change btree
+ topology - i.e. btree node splits, compactions and merges; the
+ duration measured roughly corresponds to lock held time.
+
+ \item \texttt{btree\_interior\_update\_total} \\
+ Tracks time to completion for topology changing btree updates;
+ first they have a foreground part that updates btree nodes in
+ memory, then after the new nodes are written there is a
+ transaction phase that records an update to an interior node or
+ a new btree root as well as changes to the alloc btree.
+
+ \item \texttt{data\_read} \\
+ Tracks the core read path - looking up a request in the extents
+ (and possibly also reflink) btree, allocating bounce buffers if
+ necessary, issuing reads, checksumming, decompressing, decrypting,
+ and delivering completions.
+
+ \item \texttt{data\_write} \\
+ Tracks the core write path - allocating space on disk for a new
+ write, allocating bounce buffers if necessary,
+ compressing, encrypting, checksumming, issuing writes, and
+ updating the extents btree to point to the new data.
+
+ \item \texttt{data\_promote} \\
+ Tracks promote operations, which happen when a read operation
+ writes an additional cached copy of an extent to
+ \texttt{promote\_target}. This is done asynchronously from the
+ original read.
+
+ \item \texttt{journal\_flush\_write} \\
+ Tracks writing of flush journal entries to disk, which first
+ issue cache flush operations to the underlying devices then
+ issue the journal writes as FUA writes. Time is tracked starting
+ from after all journal reservations have released their
+ references or the completion of the previous journal write.
+
+ \item \texttt{journal\_noflush\_write} \\
+ Tracks writing of non-flush journal entries to disk, which do
+ not issue cache flushes or FUA writes.
+
+ \item \texttt{journal\_flush\_seq} \\
+ Tracks time to flush a journal sequence number to disk by
+ filesystem sync and fsync operations, as well as the allocator
+ prior to reusing buckets when none that do not need flushing are
+ available.
+\end{description}
+
+\subsubsection{Internals}
+
+\begin{description}
+ \item \texttt{btree\_cache} \\
+ Shows information on the btree node cache: number of cached
+ nodes, number of dirty nodes, and whether the cannibalize lock
+ (for reclaiming cached nodes to allocate new nodes) is held.
+
+ \item \texttt{dirty\_btree\_nodes} \\
+ Prints information related to the interior btree node update
+ machinery, which is responsible for ensuring dependent btree
+ node writes are ordered correctly.
+
+ For each dirty btree node, prints:
+ \begin{itemize}
+ \item Whether the \texttt{need\_write} flag is set
+ \item The level of the btree node
+ \item The number of sectors written
+ \item Whether writing this node is blocked, waiting for
+ other nodes to be written
+ \item Whether it is waiting on a btree\_update to
+ complete and make it reachable on-disk
+ \end{itemize}
+
+ \item \texttt{btree\_key\_cache} \\
+ Prints infromation on the btree key cache: number of freed keys
+ (which must wait for a sRCU barrier to complete before being
+ freed), number of cached keys, and number of dirty keys.
+
+ \item \texttt{btree\_transactions} \\
+ Lists each running btree transactions that has locks held,
+ listing which nodes they have locked and what type of lock, what
+ node (if any) the process is blocked attempting to lock, and
+ where the btree transaction was invoked from.
+
+ \item \texttt{btree\_updates} \\
+ Lists outstanding interior btree updates: the mode (nothing
+ updated yet, or updated a btree node, or wrote a new btree root,
+ or was reparented by another btree update), whether its new
+ btree nodes have finished writing, its embedded closure's
+ refcount (while nonzero, the btree update is still waiting), and
+ the pinned journal sequence number.
+
+ \item \texttt{journal\_debug} \\
+ Prints a variety of internal journal state.
+
+ \item \texttt{journal\_pins}
+ Lists items pinning journal entries, preventing them from being
+ reclaimed.
+
+ \item \texttt{new\_stripes} \\
+ Lists new erasure-coded stripes being created.
+
+ \item \texttt{stripes\_heap} \\
+ Lists erasure-coded stripes that are available to be reused.
+
+ \item \texttt{open\_buckets} \\
+ Lists buckets currently being written to, along with data type
+ and refcount.
+
+ \item \texttt{io\_timers\_read} \\
+ \item \texttt{io\_timers\_write} \\
+ Lists outstanding IO timers - timers that wait on total reads or
+ writes to the filesystem.
+
+ \item \texttt{trigger\_journal\_flush} \\
+ Echoing to this file triggers a journal commit.
+
+ \item \texttt{trigger\_gc} \\
+ Echoing to this file causes the GC code to recalculate each
+ bucket's oldest\_gen field.
+
+ \item \texttt{prune\_cache} \\
+ Echoing to this file prunes the btree node cache.
+
+ \item \texttt{read\_realloc\_races} \\
+ This counts events where the read path reads an extent and
+ discovers the bucket that was read from has been reused while
+ the IO was in flight, causing the read to be retried.
+
+ \item \texttt{extent\_migrate\_done} \\
+ This counts extents moved by the core move path, used by copygc
+ and rebalance.
+
+ \item \texttt{extent\_migrate\_raced} \\
+ This counts extents that the move path attempted to move but no
+ longer existed when doing the final btree update.
+\end{description}
+
+\subsubsection{Unit and performance tests}
+
+Echoing into \texttt{/sys/fs/bcachefs/<uuid>/perf\_test} runs various low level
+btree tests, some intended as unit tests and others as performance tests. The
+syntax is
+\begin{quote} \begin{verbatim}
+ echo <test_name> <nr_iterations> <nr_threads> > perf_test
+\end{verbatim} \end{quote}
+
+When complete, the elapsed time will be printed in the dmesg log. The full list
+of tests that can be run can be found near the bottom of
+\texttt{fs/bcachefs/tests.c}.
+
+\subsection{Debugfs interface}
+
+The contents of every btree, as well as various internal per-btree-node
+information, are available under \texttt{/sys/kernel/debug/bcachefs/<uuid>/}.
+
+For every btree, we have the following files:
+
+\begin{description}
+ \item \textit{btree\_name} \\
+ Entire btree contents, one key per line
+
+ \item \textit{btree\_name}\texttt{-formats} \\
+ Information about each btree node: the size of the packed bkey
+ format, how full each btree node is, number of packed and
+ unpacked keys, and number of nodes and failed nodes in the
+ in-memory search trees.
+
+ \item \textit{btree\_name}\texttt{-bfloat-failed} \\
+ For each sorted set of keys in a btree node, we construct a
+ binary search tree in eytzinger layout with compressed keys.
+ Sometimes we aren't able to construct a correct compressed
+ search key, which results in slower lookups; this file lists the
+ keys that resulted in these failed nodes.
+\end{description}
+
+\subsection{Listing and dumping filesystem metadata}
+
+\subsubsection{bcachefs show-super}
+
+This subcommand is used for examining and printing bcachefs superblocks. It
+takes two optional parameters:
+\begin{description}
+ \item \texttt{-l}: Print superblock layout, which records the amount of
+ space reserved for the superblock and the locations of the
+ backup superblocks.
+ \item \texttt{-f, --fields=(fields)}: List of superblock sections to
+ print, \texttt{all} to print all sections.
+\end{description}
+
+\subsubsection{bcachefs list}
+
+This subcommand gives access to the same functionality as the debugfs interface,
+listing btree nodes and contents, but for offline filesystems.
+
+\subsubsection{bcachefs list\_journal}
+
+This subcommand lists the contents of the journal, which primarily records btree
+updates ordered by when they occured.
+
+\subsubsection{bcachefs dump}
+
+This subcommand can dump all metadata in a filesystem (including multi device
+filesystems) as qcow2 images: when encountering issues that \texttt{fsck} can
+not recover from and need attention from the developers, this makes it possible
+to send the developers only the required metadata. Encrypted filesystems must
+first be unlocked with \texttt{bcachefs remove-passphrase}.
+
+\section{ioctl interface}
+
+This section documents bcachefs-specific ioctls:
+
+\begin{description}
+ \item \texttt{BCH\_IOCTL\_QUERY\_UUID} \\
+ Returs the UUID of the filesystem: used to find the sysfs
+ directory given a path to a mounted filesystem.
+
+ \item \texttt{BCH\_IOCTL\_FS\_USAGE} \\
+ Queries filesystem usage, returning global counters and a list
+ of counters by \texttt{bch\_replicas} entry.
+
+ \item \texttt{BCH\_IOCTL\_DEV\_USAGE} \\
+ Queries usage for a particular device, as bucket and sector
+ counts broken out by data type.
+
+ \item \texttt{BCH\_IOCTL\_READ\_SUPER} \\
+ Returns the filesystem superblock, and optionally the superblock
+ for a particular device given that device's index.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_ADD} \\
+ Given a path to a device, adds it to a mounted and running
+ filesystem. The device must already have a bcachefs superblock;
+ options and parameters are read from the new device's superblock
+ and added to the member info section of the existing
+ filesystem's superblock.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_REMOVE} \\
+ Given a path to a device or a device index, attempts to remove
+ it from a mounted and running filesystem. This operation
+ requires walking the btree to remove all references to this
+ device, and may fail if data would become degraded or lost,
+ unless appropriate force flags are set.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_ONLINE} \\
+ Given a path to a device that is a member of a running
+ filesystem (in degraded mode), brings it back online.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_OFFLINE} \\
+ Given a path or device index of a device in a multi device
+ filesystem, attempts to close it without removing it, so that
+ the device may be re-added later and the contents will still be
+ available.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_SET\_STATE} \\
+ Given a path or device index of a device in a multi device
+ filesystem, attempts to set its state to one of read-write,
+ read-only, failed or spare. Takes flags to force if the
+ filesystem would become degraded.
+
+ \item \texttt{BCH\_IOCTL\_DISK\_GET\_IDX} \\
+ \item \texttt{BCH\_IOCTL\_DISK\_RESIZE} \\
+ \item \texttt{BCH\_IOCTL\_DISK\_RESIZE\_JOURNAL} \\
+ \item \texttt{BCH\_IOCTL\_DATA} \\
+ Starts a data job, which walks all data and/or metadata in a
+ filesystem performing, performaing some operation on each btree
+ node and extent. Returns a file descriptor which can be read
+ from to get the current status of the job, and closing the file
+ descriptor (i.e. on process exit stops the data job.
+
+ \item \texttt{BCH\_IOCTL\_SUBVOLUME\_CREATE} \\
+ \item \texttt{BCH\_IOCTL\_SUBVOLUME\_DESTROY} \\
+ \item \texttt{BCHFS\_IOC\_REINHERIT\_ATTRS} \\
+\end{description}
+
+\section{On disk format}
+
+\subsection{Superblock}
+
+The superblock is the first thing to be read when accessing a bcachefs
+filesystem. It is located 4kb from the start of the device, with redundant
+copies elsewhere - typically one immediately after the first superblock, and one
+at the end of the device.
+
+The \texttt{bch\_sb\_layout} records the amount of space reserved for the
+superblock as well as the locations of all the superblocks. It is included with
+every superblock, and additionally written 3584 bytes from the start of the
+device (512 bytes before the first superblock).
+
+Most of the superblock is identical across each device. The exceptions are the
+\texttt{dev\_idx} field, and the journal section which gives the location of the
+journal.
+
+The main section of the superblock contains UUIDs, version numbers, number of
+devices within the filesystem and device index, block size, filesystem creation
+time, and various options and settings. The superblock also has a number of
+variable length sections:
+
+\begin{description}
+ \item \texttt{BCH\_SB\_FIELD\_journal} \\
+ List of buckets used for the journal on this device.
+
+ \item \texttt{BCH\_SB\_FIELD\_members} \\
+ List of member devices, as well as per-device options and
+ settings, including bucket size, number of buckets and time when
+ last mounted.
+
+ \item \texttt{BCH\_SB\_FIELD\_crypt} \\
+ Contains the main chacha20 encryption key, encrypted by the
+ user's passphrase, as well as key derivation function settings.
+
+ \item \texttt{BCH\_SB\_FIELD\_replicas} \\
+ Contains a list of replica entries, which are lists of devices
+ that have extents replicated across them.
+
+ \item \texttt{BCH\_SB\_FIELD\_quota} \\
+ Contains timelimit and warnlimit fields for each quota type
+ (user, group and project) and counter (space, inodes).
+
+ \item \texttt{BCH\_SB\_FIELD\_disk\_groups} \\
+ Formerly referred to as disk groups (and still is throughout the
+ code); this section contains device label strings and records
+ the tree structure of label paths, allowing a label once parsed
+ to be referred to by integer ID by the target options.
+
+ \item \texttt{BCH\_SB\_FIELD\_clean} \\
+ When the filesystem is clean, this section contains a list of
+ journal entries that are normally written with each journal
+ write (\texttt{struct jset}): btree roots, as well as filesystem
+ usage and read/write counters (total amount of data read/written
+ to this filesystem). This allows reading the journal to be
+ skipped after clean shutdowns.
+\end{description}
+
+\subsection{Journal}
+
+Every journal write (\texttt{struct jset}) contains a list of entries:
+\texttt{struct jset\_entry}. Below are listed the various journal entry types.
+
+\begin{description}
+ \item \texttt{BCH\_JSET\_ENTRY\_btree\_key} \\
+ This entry type is used to record every btree update that
+ happens. It contains one or more btree keys (\texttt{struct
+ bkey}), and the \texttt{btree\_id} and \texttt{level} fields of
+ \texttt{jset\_entry} record the btree ID and level the key
+ belongs to.
+
+ \item \texttt{BCH\_JSET\_ENTRY\_btree\_root} \\
+ This entry type is used for pointers btree roots. In the current
+ implementation, every journal write still records every btree
+ root, although that is subject to change. A btree root is a bkey
+ of type \texttt{KEY\_TYPE\_btree\_ptr\_v2}, and the btree\_id
+ and level fields of \texttt{jset\_entry} record the btree ID and
+ depth.
+
+ \item \texttt{BCH\_JSET\_ENTRY\_clock} \\
+ Records IO time, not wall clock time - i.e. the amount of reads
+ and writes, in 512 byte sectors since the filesystem was
+ created.
+
+ \item \texttt{BCH\_JSET\_ENTRY\_usage} \\
+ Used for certain persistent counters: number of inodes, current
+ maximum key version, and sectors of persistent reservations.
+
+ \item \texttt{BCH\_JSET\_ENTRY\_data\_usage} \\
+ Stores replica entries with a usage counter, in sectors.
+
+ \item \texttt{BCH\_JSET\_ENTRY\_dev\_usage} \\
+ Stores usage counters for each device: sectors used and buckets
+ used, broken out by each data type.
+\end{description}
+
+\subsection{Btrees}
+
+\subsection{Btree keys}
+
+\begin{description}
+ \item \texttt{KEY\_TYPE\_deleted}
+ \item \texttt{KEY\_TYPE\_whiteout}
+ \item \texttt{KEY\_TYPE\_error}
+ \item \texttt{KEY\_TYPE\_cookie}
+ \item \texttt{KEY\_TYPE\_hash\_whiteout}
+ \item \texttt{KEY\_TYPE\_btree\_ptr}
+ \item \texttt{KEY\_TYPE\_extent}
+ \item \texttt{KEY\_TYPE\_reservation}
+ \item \texttt{KEY\_TYPE\_inode}
+ \item \texttt{KEY\_TYPE\_inode\_generation}
+ \item \texttt{KEY\_TYPE\_dirent}
+ \item \texttt{KEY\_TYPE\_xattr}
+ \item \texttt{KEY\_TYPE\_alloc}
+ \item \texttt{KEY\_TYPE\_quota}
+ \item \texttt{KEY\_TYPE\_stripe}
+ \item \texttt{KEY\_TYPE\_reflink\_p}
+ \item \texttt{KEY\_TYPE\_reflink\_v}
+ \item \texttt{KEY\_TYPE\_inline\_data}
+ \item \texttt{KEY\_TYPE\_btree\_ptr\_v2}
+ \item \texttt{KEY\_TYPE\_indirect\_inline\_data}
+ \item \texttt{KEY\_TYPE\_alloc\_v2}
+ \item \texttt{KEY\_TYPE\_subvolume}
+ \item \texttt{KEY\_TYPE\_snapshot}
+ \item \texttt{KEY\_TYPE\_inode\_v2}
+ \item \texttt{KEY\_TYPE\_alloc\_v3}
+\end{description}
+
+\end{document}
projects / bcachefs-tools-debian / commitdiff