mkosi is over-eager to use "cp --preserve=xattr", fails in cross-filesystem cases

[gdonval]

mkosi commit the issue has been seen with

main

Used host distribution

archlinux

Used target distribution

ubuntu

Linux kernel version used

6.14.6-arch1-1

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

WorkspaceDirectory set to XFS-supported directory (here /tmp/mkosi.workspace). All other mkosi dirs on BTRFS-supported directory.

...
‣ Copying repository metadata
cp: setting attributes for '/work/tmp/mkosi-gdonval-workspace/mkosi/mkosi-metadata-lwbvcpek/cache/apt/srcpkgcache.bin': Operation not supported
...
cp: setting attributes for '/work/tmp/mkosi-gdonval-workspace/mkosi/mkosi-metadata-lwbvcpek/cache/apt': Operation not supported
‣ "cp --recursive --no-dereference --preserve=mode,links,timestamps,ownership,xattr --reflink=auto --copy-contents '/work/home/gdonval/.cache/mkosi/ubuntu~noble~x86-64/cache/apt' /work/tmp/mkosi-gdonval-workspace/mkosi/mkosi-metadata-lwbvcpek/cache/apt --no-target-directory" returned non-zero exit code 1.

The reason it happens it because original files have attrs that cannot be copied, e.g:

$ lsattr -a /home/gdonval/Documents/Work/RVMI/Services/UbuntuRVMI/mkosi.cache/ubuntu~noble~x86-64~main.metadata.cache/cache/apt/srcpkgcache.bin
--------c------------- /home/gdonval/Documents/Work/RVMI/Services/UbuntuRVMI/mkosi.cache/ubuntu~noble~x86-64~main.metadata.cache/cache/apt/srcpkgcache.bin
$ lsattr -a /tmp/mkosi-gdonval-workspace/mkosi/mkosi-metadata-lwbvcpek/cache/apt/srcpkgcache.bin
---------------------- /tmp/mkosi-gdonval-workspace/mkosi/mkosi-metadata-lwbvcpek/cache/apt/srcpkgcache.bin

That c is invalid on XFS.

Used mkosi config

[Distribution]
Distribution=ubuntu
Release=noble
Repositories=universe multiverse

[Build]
WorkspaceDirectory=/tmp/mkosi.workspace

[Output]
Format=disk

mkosi output

[behrmann]

I've had this alternative to #3692 flying around for a while. Would this work for you?

[labichn]

BTW @gdonval, @behrmann, this copy issue doesn't seem to be XFS-specific, cp ... --preserve=xattr fails on a cross-device copy between two BTRFS filesystems (/ and a LUKS-device-backed BTRFS homed) when running latest main as root on my system.

I suspect I'd be able to work around the issue with MKOSI__CP_PRESERVE_ATTR from @behrmann's branch, but it seems like this is just broken, no?

cat <<'EOF' > mkosi.conf
[Distribution]
Distribution=arch
[Output]
Format=directory
ImageId=test
[Content]
Packages=systemd
EOF

cat <<'EOF' > mkosi.postinst
#!/usr/bin/bash
lsattr -d $BUILDROOT/var/log/journal
EOF

chmod a+x mkosi.postinst

sudo mkosi  # Note: does not fail if not running as root
# ...
# ‣  Running postinstall script /home/nick/tmp/mkosi.postinst…
# ---------------C------ /buildroot/var/log/journal
# ...
# ‣ Could not rename /var/tmp/mkosi-workspace-m5t2e24a/staging/test to /home/nick/tmp/test as they are located on different devices, falling back to copying
ERROR: Could not create subvolume: Invalid cross-device link
cp: setting attributes for '/work/home/nick/tmp/test/var/log/journal': Operation not supported
# ‣ "cp --recursive --no-dereference --preserve=mode,links,timestamps,ownership,xattr --reflink=auto --copy-contents /work/var/tmp/mkosi-workspace-m5t2e24a/staging/test /work/home/nick/tmp/test --no-target-directory" returned non-zero exit code 1.

[behrmann]

Do you know which combinations don't work? xfs <-> btrfs, btrfs <-> btrfs

[DaanDeMeyer]

@gdonval @labichn This is with cp from coreutils right? I cloned the codebase and can't figure out for the life of me where they handle file attributes at all.

[DaanDeMeyer]

If it is from coreutils, can I get the exact coreutils version?

[gdonval]

$ cp --version
cp (GNU coreutils) 9.7
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Torbjörn Granlund, David MacKenzie, and Jim Meyering.

I'll try to create a reproducer.

[DaanDeMeyer]

Ah I wonder if this coming from inside gnulib

[DaanDeMeyer]

Doesn't seem to coming from there either. I'm a bit stuck until I know where this error message is coming from inside coreutils. I need the source code to figure out what triggers this and how we could potentially skip it. If they tie file attributes copying to xattr copying that'd be a rather questionable decision if you ask me.

[gdonval]

If they tie file attributes copying to xattr copying that'd be a rather questionable decision if you ask me

It looks like that's exactly what happens.

workspace="/tmp/cp_attr"

mkdir -p ${workspace}/{xfs,btrfs}
truncate -s 512M ${workspace}/xfs.img
truncate -s 512M ${workspace}/btrfs.img

mkfs.xfs ${workspace}/xfs.img
mkfs.btrfs ${workspace}/btrfs.img

run0 -D ${workspace} sh -c "
    mount -o loop xfs.img xfs
    mount -o loop btrfs.img btrfs
    mkdir btrfs/foo
    touch btrfs/foo/{bar,baz}
    chattr +c btrfs/foo/baz

    lsattr btrfs/foo/
    
    echo 'copying btrfs/foo/bar -> xfs/bar'
    cp --preserve=xattr btrfs/foo/bar xfs/
    echo 'copying btrfs/foo/baz -> xfs/baz'
    cp --preserve=xattr btrfs/foo/baz xfs/ || echo 'Failed'

    umount btrfs
    umount xfs
"

rm ${workspace}/{xfs,btrfs}.img
rmdir ${workspace}/{xfs,btrfs}

Output:

---------------------- btrfs/foo/bar
--------c------------- btrfs/foo/baz
copying btrfs/foo/bar -> xfs/bar
copying btrfs/foo/baz -> xfs/baz
cp: setting attributes for 'xfs/baz': Operation not supported
Failed

Naturally, if I remove c on btrfs/foo/baz, the copy goes through so indeed cp seems to conflate those two kinds of xattrs.

[labichn]

If it is from coreutils, can I get the exact coreutils version?

It is: cp (GNU coreutils) 9.7, from Arch's core/coreutils 9.7-1.

I tried to cut down my laptop's image to a minimal reproducer, but can't get this same error to occur in QEMU.

[gdonval]

but it's not a terminating error and mkosi exits successfully:

Yeah that's fine. Rename is tried and if it fails it falls back to copy.

I haven't observed any errors with any other filesystems other than cross-device btrfs<->btrfs

Does it actually stop mkosi or does it only display that btrfs error?

That's where it happens:

mkosi/mkosi/tree.py

Lines 228 to 239 in 5e739ef

	try:
	src.rename(dst)
	except OSError as e:
	if e.errno != errno.EXDEV:
	raise e
	logging.info(
	f"Could not rename {src} to {dst} as they are located on different devices, "
	"falling back to copying"
	)
	copy_tree(src, dst, use_subvolumes=use_subvolumes, sandbox=sandbox)
	rmtree(src, sandbox=sandbox)

So indeed it's not supposed to stop mkosi and should have no negative consequence whatsoever. It's telling you that it cannot do something efficient (rename) because source and destination are on different partitions. It's just that the corresponding btrfs CLI error seeps through and shows up in the terminal.

This is also unrelated to xattr.

[labichn]

but it's not a terminating error and mkosi exits successfully:

Yeah that's fine. Rename is tried and if it fails it falls back to copy.

Oh sure, not the first line, I meant the ERROR: Could not create subvolume: Invalid cross-device link is non-terminating, only displayed.

Glad to clip out the failure to reproduce to a gist, it's more noise than useful.