rust: update bindgen to 0.69.4; remove custom type modifications
This updates rust-bindgen to version 0.69.4 which includes the patch 199bee441ad0: "try to avoid #[repr(packed)] when align is needed". With
this patch, bindgen generates code that is both ABI-correct and can be
compiled by rustc, for 3 bcachefs types:
- bkey
- bch_extent_crc32
- bch_extent_ptr
This allows us to remove the custom treatment for these three types.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Now that the bcachefs tool unconditionally includes the mount parts
(or more correctly, you cannot build it at all if you don't have Rust),
we can call copy_exec on mount.bcachefs, to get the symlink installed.
In particular, this helps with mounting UUID mounts as /.
See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1060411
for the remaining parts needed in initramfs-tools itself.
Ryan Lahfa [Sat, 27 Jan 2024 03:23:20 +0000 (04:23 +0100)]
feat(rust/wrappers): init `BcachefsHandle`
We propose a simple low-level wrapper which can perform various subvolume-related operations
as an example for the API surface.
This will be used in an upcoming commit to migrate the subvolume CLI fully to Rust.
The API design is the following:
- `BcachefsHandle` is meant as a low level handle to carry around whenever you need a filesystem handle
to send ioctl to.
- it possess type-safe operations
Type safe operations are handled by having type safe wrappers for ioctl commands
*and* their payloads.
We assume that all ioctl payloads only use *one* argument, this can easily be changed if needed.
When the handle goes out of scope, we automatically close it à la C++ RAII.
use upstream bindgen; fix packed and aligned types
bcachefs-tools has been using a patched bindgen to work around a
limitation of rustc that prevents compiling structs with
both #[repr(packed(N)] and #[repr(align(N)] attributes. The patch: e8168ceda507 "codegen: Don't generate conflicting packed() and align()
representation hints." discards the "align" attribute in cases where
bindgen produces a type with both.
This may be correct for some types, but it turns out that for each
bcachefs type with this problem, keeping the "align" attribute and
discarding the "packed" attribute generates a type with the same ABI as
the original C type.
This can be tested automatically by running:
$ cargo test --manifest-path bch_bindgen/Cargo.toml
in the bcachefs-tools tree.
There has been pressure recently to start using upstream bindgen; both
externally, from distribution maintainers who want to build
bcachefs-tools with standard dependencies, and internally, in order to
enable using Rust for bcachefs in-kernel.
This patch updates bcachefs-tools to use upstream bindgen. It works
around the rustc limitation with a post-processing step in the bindgen
build that adjusts the attributes to include "#[repr(C, align(N))]" and
exclude #[repr(packed(N)] only for the 4 types that need it. It also
updates bch_bindgen to format the code with prettyplease so that this
will work even in environments with rustfmt installed.
Some types that had been manually implemented in
bch_bindgen/src/bcachefs.rs are now automatically generated by bindgen,
so that they will be covered by the ABI compatibility testing mentioned
above.
I intentionally targeted the post-processing to the exact 4 types with
the issue currently, so that any changes to bcachefs that result in this
issue appearing for a new type will require manual intervention. I
figured any such changes should require careful consideration.
Ideally, bindgen can be updated to handle situations where "align(N)"
is needed and "packed(N)" can be safely discarded. If a patch for this
is accepted into bindgen, the post-processing hack can be removed.
I update the minimum Rust version to 1.70 as this is needed to build
recent versions of some dependencies.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
remove bch_bindgen/Cargo.lock from version control
This is redundant with the root level Cargo.lock. Any changes made to
the bch_bindgen dependencies will be duplicated in both Cargo.lock
files. Removing this from version control will reduce the noise in the
git diffs for such changes.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
When bcachefs was a C program that had some functions implemented in
Rust, it was necessary to make a static library containing the Rust
functions available for the C program to link.
Now that bcachefs is a Rust program, that library is no longer needed.
Instead, the Rust executable links in libbachefs.a.
This patch updates the crate structure to reflect that. The command
functions are moved into their own module.
There could be a need to create a "libbachefs-tools" library in the
future that exposes an API for bcachefs functionality to other
userspace programs. That will be a different, external API as opposed to
the previous library functions which were an internal API for the
bcachefs tool itself.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
The only use for itertools is in parse_mount_options() where we take a
vector, convert it to iterator and then join it. Instead, we can join
the vector directly.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
The memmove() in pop_cmd() reads and writes beyond the end of argv.
This is basically harmless in the current C program; the environment
variable list immediately follows argv so all this does is unnecessarily
copy the beginning of that list.
However, this will become problematic once we start calling C functions
like fs_cmds() from Rust code. Then argv will be a Vec<String> (as
*mut *mut i8) and the memory layout will be different--in particular,
I don't think we can assume that a Vec<String> will be NULL-terminated
like argv always is--, meaning the invalid write could lead to heap
corruption.
Also, it doesn't look like full_cmd ever gets used after calling
pop_cmd() so I'm removing it here since it looks unneeded to me.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
block_size is in bytes, not sectors, so when calling round_up(),
we could start rounding up by a way too large size and then overflow
outside the area that migrate allocated for us.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
The bcachefs sub-commands that are implemented in Rust (completions,
list, and mount) had separate entrypoints and thus had some differences
in behavior.
This introduces a common entry point for the Rust sub-commands. This
reduces duplicate boilerplate code like parsing argv and setting up
logging, and will facilitate converting more sub-commands to Rust in
the future.
An immediate benefit is that this fixes an issue with `bcachefs list`
not reporting errors:
before:
$ bcachefs list /dev/typo
$ echo $?
0
after:
$ bcachefs list /dev/typo
ERROR - bcachefs_rust::cmd_list: Fatal error: "No such file or directory"
$ echo $?
1
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Neal Gompa [Sun, 24 Dec 2023 13:34:16 +0000 (08:34 -0500)]
Makefile, fsck: Use libexec instead of lib
These are executables that need to be installed in a non-path location.
Most distributions now install these into /usr/libexec, and the
path variable for this is LIBEXECDIR, so use that instead.
Kent Overstreet [Sun, 24 Dec 2023 00:37:04 +0000 (19:37 -0500)]
cmd_fsck: -k, run fsck in kernel
This adds a new option to cmd_fsck for using the kernel implementation
of fsck instead of userspace, via the BCH_IOCTL_FSCK_OFFLINE ioctl.
This isn't intended for normal usage - mainly for testing and debugging
purposes, and for when the kernel version of bcachefs better matches the
on disk format version.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Specifically, do not recursively expand $(CFLAGS) because this leads to
repeatedly performing compile tests (e. g. cc-disable-warning) on every
recipe execution.
Without (nproc=32):
```
$ time env -i PATH=/usr/bin BCACHEFS_FUSE=1 NO_RUST=1 make -j$(nproc)
<...>
[LD] bcachefs
72,48s user 11,29s system 190% cpu 44,036 total
```
With:
```
$ time env -i PATH=/usr/bin BCACHEFS_FUSE=1 NO_RUST=1 make -j$(nproc)
<...>
[LD] bcachefs
66,79s user 5,17s system 1955% cpu 3,679 total
```
This adds on to 446ec6b092cc ("docs: mention that libclang is
required"). That commit updated the dependency list for Debian. This
adds the packages needed to get llvm-config for arch and fedora.
Signed-off-by: Thomas Bertschinger <tahbertschinger@gmail.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Kent Overstreet [Wed, 20 Dec 2023 00:17:19 +0000 (19:17 -0500)]
cmd_migrate: fix write_data()
passing a parent closure is totally unnecessary when using
BCH_WRITE_SYNC - and pops an assert in debug mode, because the write
path no longer supports delivering completions that way.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 14:31:40 +0000 (14:31 +0000)]
bcachefs-tools: Disable -Wgnu-variable-sized-type-not-at-end for clang
clang's default warnings include -Wgnu-variable-sized-type-not-at-end,
producing a lot of false alarms about the GNU extension for libbcachefs.
Disable these using cc-disable-warning when building with clang.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Chris Webb [Sat, 9 Dec 2023 14:15:08 +0000 (14:15 +0000)]
bcachefs-tools: Guard the __struct_group() #define in kernel.h
Our include/linux/byteorder.h uses the system <asm/byteorder.h>
because we don't provide our own in include/asm. This then pulls in
system <linux/stddef.h> which defines __struct_group. That definition
collides with the unconditional #define __struct_group in our local
include/linux/kernel.h, provoking preprocessor warnings.
Signed-off-by: Chris Webb <chris@arachsys.com> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>