Specifically, do not recursively expand $(CFLAGS) because this leads to
repeatedly performing compile tests (e. g. cc-disable-warning) on every
recipe execution.
Without (nproc=32):
```
$ time env -i PATH=/usr/bin BCACHEFS_FUSE=1 NO_RUST=1 make -j$(nproc)
<...>
[LD] bcachefs
72,48s user 11,29s system 190% cpu 44,036 total
```
With:
```
$ time env -i PATH=/usr/bin BCACHEFS_FUSE=1 NO_RUST=1 make -j$(nproc)
<...>
[LD] bcachefs
66,79s user 5,17s system 1955% cpu 3,679 total
```
This adds on to 446ec6b092cc ("docs: mention that libclang is
required"). That commit updated the dependency list for Debian. This
adds the packages needed to get llvm-config for arch and fedora.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Kent Overstreet [Wed, 20 Dec 2023 00:17:19 +0000 (19:17 -0500)]
cmd_migrate: fix write_data()
passing a parent closure is totally unnecessary when using
BCH_WRITE_SYNC - and pops an assert in debug mode, because the write
path no longer supports delivering completions that way.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 14:31:40 +0000 (14:31 +0000)]
bcachefs-tools: Disable -Wgnu-variable-sized-type-not-at-end for clang
clang's default warnings include -Wgnu-variable-sized-type-not-at-end,
producing a lot of false alarms about the GNU extension for libbcachefs.
Disable these using cc-disable-warning when building with clang.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 14:15:08 +0000 (14:15 +0000)]
bcachefs-tools: Guard the __struct_group() #define in kernel.h
Our include/linux/byteorder.h uses the system <asm/byteorder.h>
because we don't provide our own in include/asm. This then pulls in
system <linux/stddef.h> which defines __struct_group. That definition
collides with the unconditional #define __struct_group in our local
include/linux/kernel.h, provoking preprocessor warnings.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 12:50:31 +0000 (12:50 +0000)]
bcachefs-tools: Avoid glibc-specific mallinfo() in shrinker
Before 326d7c1, the shrinker used freeram and totalram from a struct
sysinfo (constructed from /proc/meminfo) to target 25% free physical
memory. As well as the slowness of repeatedly reading /proc/meminfo,
this was a problem as freeram rises when the system starts to swap.
We don't want swapping to reduce our estimate of memory pressure.
To work around this, in 326d7c1 the shrinker started to use the total
allocated heap from a glibc-specific interface mallinfo2(), aiming to
shrink such that our heap is less than 80% of physical memory, unless
overall free memory is less than 6% so that becomes the determining factor.
Unfortunately, a sign error in the calculation means this heuristic
never worked. It would shrink aggressively when the process was small,
and not at all when the process grew beyond 80% of physical RAM. Only the
fallback test ensuring the free physical RAM doesn't fall below 6% would
actually kick in under memory pressure. It also breaks portability to
anything other than recent glibc.
Later, in 2440469 the mallinfo2() was replaced with the older mallinfo()
to improve compatibility with older glibc. This is even more problematic:
it's still not portable but also struct mallinfo has (signed) int fields
which overflow for large processes on 32-bit machines with a 3G/1G split.
Rather than trying to use libc-specific debug interfaces and our own heap
to inform the shrinker, use the information about free and total swap
we already have from sysinfo(2) to explicitly compensate for swapping
in our estimate of free physical memory. Target free memory of 6% of
physical RAM adjusted for zero swap use when calculating the pressure
on the shrinker, based on the effective behaviour of 326d7c1 in practice
given the sign error.
As well as fixing portability to non-glibc systems, this loosens the
assumption that we are the only process using significant memory when
setting the shrinker target. It wouldn't be unreasonable to run two
fsck jobs against independent devices on a large RAM machine and want to
balance physical RAM between them.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 12:49:12 +0000 (12:49 +0000)]
bcachefs-tools: Use sysinfo(2) directly to implement si_meminfo()
Use a single sysinfo(2) call to fill out struct sysinfo instead of
multiple libc sysconf(3) requests, which will only make sysinfo(2) calls
internally anyway. This also enables us to access other struct sysinfo
fields, not just the three filled-out previously.
As we provide our own definition of struct sysinfo in include/linux/mm.h
to match the kernel, which is not guaranteed to align with the definition
libc provides in <sys/sysinfo.h>, use syscall(SYS_sysinfo, ...) directly
instead of the libc wrapper.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Tero Roponen [Fri, 8 Dec 2023 08:10:52 +0000 (10:10 +0200)]
bcachefs-tools: make 'all' the default Makefile goal
"fsck/bcachefsck_all:" coming before "all:" in the Makefile
causes the bcachefs binary not being built by default. Fix
this by explicitly setting the .DEFAULT_GOAL=all.
Signed-off-by: Tero Roponen <tero.roponen@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Darrick J. Wong [Tue, 5 Dec 2023 02:43:31 +0000 (18:43 -0800)]
fsck: add a service to schedule and run bcachefsck@ services
Create a script that looks for mounted bcachefs filesystems and tries to
run as many of them in parallel as possible without two different
scrubbers hitting the same low level block device.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Darrick J. Wong [Tue, 5 Dec 2023 02:43:31 +0000 (18:43 -0800)]
fsck: add systemd service definitions for automatic online service
Add some systemd service files so that bcachefs can automatically fsck
mounted filesystems in the background. Hopefully with minimal
disruption to frontend operations.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Multiple small fuse fixes for write, api, and hardlink
* changed in Makefile to use recomended fuse api 35;
* force foreground mode, because fuse_daemonize is forking after initalization other bcachefs threads;
* fix fuse_link, there where mistype in fuse_log (the paremeter need to be inum.inum);
* The write_aligned is fixed, by wrapping struct bch_write_op into container with single closure and using end_io callback to release
that closure.
Kent Overstreet [Tue, 5 Dec 2023 23:28:14 +0000 (18:28 -0500)]
Disable shrinker thread shutdown
We seem to be hitting a rare crash in the exit path of fsck - when
shutting down the shrinker thread. Disable exiting the shrinker thread
as a workaround.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Darrick J. Wong [Tue, 5 Dec 2023 00:44:29 +0000 (19:44 -0500)]
misc: don't allow udisks to automount bcachefs filesystems with no prompt
The unending stream of syzbot bug reports and overwrought filing of CVEs
for corner case handling (i.e. things that distract from actual user
complaints) in XFS has generated all sorts of of overheated rhetoric
about how every bug is a Serious Security Issue(tm) because anyone can
craft a malicious filesystem on a USB stick, insert the stick into a
victim machine, and mount will trigger a bug in the kernel driver that
leads to some compromise or DoS or something.
I thought that nobody would be foolish enough to automount an XFS
filesystem. What a fool I was! It turns out that udisks can be told
that it's okay to automount things, and then GNOME will do exactly that.
Including mounting mangled XFS filesystems! Same with bcachefs!
<delete angry rant about poor decisionmaking and armchair fs developers
blasting us on X while not actually doing any of the work>
Turn off /this/ idiocy by adding a udev rule to tell udisks not to
automount bcachefs filesystems.
This will not stop a logged in user from unwittingly inserting a
malicious storage device and pressing [mount] and getting breached.
This is not a substitute for a thorough audit. This is not a substitute
for lklfuse. This does not solve the general problem of in-kernel fs
drivers being a huge attack surface. I just want to give Kent a break
from some of the oceans of bu******.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Brian Foster [Mon, 27 Nov 2023 18:31:10 +0000 (13:31 -0500)]
bcachefs-tools: don't skip stale superblock wipe in force mode
Even though the blkid wipe has been fixed to detect and wipe old
superblocks, we still incorrectly skip the wipe when in force mode.
Update the force logic in open_for_format() to bypass the user
request and otherwise proceed with the wipe.
Signed-off-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
This fixes the following build failure on Debian bookworm:
error: failed to run custom build command for `clang-sys v1.6.1`
Caused by:
process didn't exit successfully: `/home/minoru/src/bcachefs-tools/rust-src/target/release/build/clang-sys-df95f6d1266be773/build-script-build` (exit status: 101)
--- stdout
cargo:warning=could not execute `llvm-config` one or more times, if the LLVM_CONFIG_PATH environment variable is set to a full path to valid `llvm-config` executable it will be used to try to find an instance of `libclang` on your system: "couldn't execute `llvm-config --prefix` (path=llvm-config) (error: No such file or directory (os error 2))"
--- stderr
thread 'main' panicked at /home/minoru/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clang-sys-1.6.1/build/dynamic.rs:206:45:
called `Result::unwrap()` on an `Err` value: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Daniel Hill [Mon, 20 Nov 2023 12:43:52 +0000 (01:43 +1300)]
improve kmalloc performance
Reading from /proc/meminfo is really slow
We don't want to start swapping to disk.
Deceptively, memory available goes up when we start to swap to disk making
performance even worse.
To mitigate this:
1. replace reading from meminfo with proper system calls.
2. attempt to lock allocations in physical memory space.
3. check our own allocated memory instead of available memory.
4. still check available memory in the off chance we're trying to play
nice with other apps.
Signed-off-by: Daniel Hill <daniel@gluo.nz> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
When playing around with comparing some fstests results on different
filesystems, I noticed that a 'bcachefs format' of a previously
btrfs-formatted device still continued to mount as btrfs. The reason
for this is that the blkid wipe invoked via open_for_format() is not
working correctly. blkid_do_wipe() depends on the "SBMAGIC[_OFFSET]"
values to do any work, and the associated superblock magic flag is
not enabled on the probe.
Set the probe flags to explicitly enable the values the bcachefs
code depends on in the probe. This includes the type, label and
superblock magic information.
There are also a couple quirks in the libblkid code that might be
worth noting. One is that the superblock enablement and flag setting
functions appear hardcoded to return zero, so we just combine the
error checks. Second, while blkid_do_wipe() can return an error, it
actually doesn't in the scenario being addressed here because it
doesn't seem to distinguish between the values being absent because
nothing was found by the probe or because the values weren't enabled
in the first place. Regardless, add an error check here in the event
the wipe does explicitly fail for some unexpected reason.
Signed-off-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
prior to this patch, on certain platforms (ie: armv7l), compilation fails due
to atomic64_sub_return_release not being defined here. Ensure that the atomics
header is pulled in, and ensure that it is available in all cases, regardless
of whether ATOMIC64_SPINLOCK is defined.
Signed-off-by: Nicholas Sielicki <linux@opensource.nslick.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
libc proper treats mount flags as an unsigned long, which is usually u64,
except when it isn't. When preparing mount flags, use the libc::c_ulong type
instead of u64 to allow for this.
This fixes compiling this file under armv7l.
Signed-off-by: Nicholas Sielicki <linux@opensource.nslick.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Kent Overstreet [Tue, 7 Nov 2023 01:47:48 +0000 (20:47 -0500)]
Avoiding picking different bucket sizes for different devices
Erasure coding can't create stripes across devices with different bucket
sizes - therefore, format shouldn't pick different bucket sizes for
different devices.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Kent Overstreet [Tue, 7 Nov 2023 00:01:15 +0000 (19:01 -0500)]
cmd_format: Check for device options after device arguments
It's a common user error to specify device specific options at the end
of a format command, and then not have them apply to any devices - add a
check for this.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
projects / bcachefs-tools-debian / log