QEMU execution engine

[staticfloat]

It could potentially be useful to have a QEMU-based execution engine, for users that want something lighter weight than Docker on non-Linux machines. We used to have a QEMU backend in BinaryBuilder, which may be a good place to start. There were a few added complexities to this setup, that will need to be mapped over to the Sandbox.jl way of doing things:

1. We originally needed to use Keno's QEMU fork to get better 9p performance for read/write mounts. I assume that since it's been a few years, these commits have made it into a QEMU release and we can probably just use mainline QEMU, but it's worth checking.

2. We needed to create an init replacement to do things like set up mounts, define environment variables, etc... inside of the QEMU environment. We did this by adding an "init mode" to our sandbox.c helper. Combining these two very different execution types (user namespaces and qemu init) allowed us to share some code, but it needlessly complicated the tool, and I'd much rather have separate sandbox and qemu_init binaries. If we need to, we can share some utility .c files between the two projects. The init mode did things like mount /dev, setup VM networking, and cleanup all state before powering off the emulator.

3. Although QEMU helpfully maps stdin and stdout for us, there's not an easy way for us to get command-line arguments or environment variables into the inside of the emulator, so we create our own communications channel by telling QEMU to create a fake serial device on the inside, and bind it to a unix domain socket on the outside, then serialize our metadata and transmit it to the init replacement on the inside. The init replacement also needs to receive that information and deal with it accordingly.

It's entirely possible that there are easier ways to deal with this complexity now than we had back then, but this is how we used to do this. The number of features that Sandbox.jl supports are even greater than what BB used to support, and some of them are going to require some thought. The feature list is basically the list of things you can configure in a SandboxConfig. The main things to figure out are:

• read_only_maps: In the past, we provided .squashfs compiler shards precisely because QEMU supported them. Since other executors such as Docker don't support those, we may need to come up with a solution that works well for both. It's possible that the 9p filesharing has become much faster now than it was before, testing will be needed.
• persist: how to best persist changes from one QEMU invocation to another. One possibility is to have a single QEMU instance running as long as the Executor object lives, then launch processes inside it by communicating over the serial socket. Another possibility is to have a host-backed filestore for all modifications made during execution, then map that in for subsequent executions.
• multiarch: This could be as simple as mapping to different QEMU host binaries, although we will of course need to provide different Linux and init bundles for each target architecutre.

[fingolfin]

Re the QEMU fork by @Keno, I don't see any traces of that in https://github.com/qemu/qemu -- did someone actually ever try to submit it upstream?

[giordano]

did someone actually ever try to submit it upstream?

It was mentioned on Slack a few days ago, good to share it also here:

• qemu: add 9p virtfs support on Darwin NixOS/nixpkgs#122420 (comment)
• https://gitlab.com/wwcohen/qemu/-/commits/9p-darwin