NAME

fuguvm — install and manage OpenBSD virtual machines

SYNOPSIS

DESCRIPTION

fuguvm installs and manages OpenBSD virtual machines under QEMU. It downloads the install media, creates the disk, answers the installer over the serial console, and caches the installed disk. It then drives the lifecycle of the guest: boot, wait, ssh, file transfer, snapshot, and shutdown. One command takes a project from nothing to a guest that accepts ssh(1), and a second run of that command changes nothing.

The host does not have to run OpenBSD. A Linux or Darwin host that runs QEMU is sufficient, and fuguvm selects the accelerator to match the host.

A project is a directory that holds a .fuguvmrc file and a .fuguvm/ directory. Without --project, fuguvm searches up from the current directory for the file. The directory that holds it is the project root. See CONFIGURATION.

The options are as follows:

--emulate

Run the guest under TCG software emulation, whatever the host can do.

-h, --help

Print the usage message and exit.

--project dir

Use dir as the project root, and do not search for one.

-q, --quiet

Do not print informational messages. The data lines of status, disk info, image export, mirror fetch and snapshot list --names still print.

-v, --verbose

Print debug messages as well.

--vm name

Operate on the named guest. The default is the default_vm setting, or default.

The commands are as follows:

init [dir]

Initialize a project in dir, which defaults to the current directory. The command writes a .fuguvmrc with the three project settings, a .fuguvm/ directory with the vms/ and state/ subdirectories, and the declaration of the default guest. It does nothing when the project is initialized already.

up [--no-cache]

Make sure that the guest runs. The command restores the disk from the image cache, or downloads the install media and installs the guest, and then starts it. It installs the ssh_pubkey when the guest does not hold it yet. See INSTALLATION.

--no-cache skips both the restore and the save of the image cache for this run, and overrides the image_cache setting. The guest then installs from scratch onto a standalone disk.

down

Stop the mirror proxy and the guest, and record a clean shutdown.

destroy

Stop the guest and delete its disk and state.

start

Start the guest in the background. The command needs an existing disk, refuses a running guest with status 5, and does not start the proxy.

stop [-f, --force]

Stop the guest. The shutdown syncs the guest filesystems, powers the guest off through ACPI, and forces the process to stop only after that. Every step is bounded. With -f, the command kills the process at once.

status [key]

Print the state of the guest to standard output, as one key: value line for each key, in sorted key order. The keys are accel, arch, bind_address, console_port, disk_exists, installed, name, pid, proxy_url, ssh_port and state. An empty value means “not now”: a stopped guest has no pid, no proxy_url, and no port for a directive of auto.

With key, print the bare value of that one key. An unknown key exits with status 2, and the message names every key.

wait [--timeout seconds]

Block until the guest accepts an SSH connection. The default timeout is 120 seconds, and a timeout exits with status 7.

ssh [--] [command [argument ...]]

Open an interactive SSH session to the guest as root, or run one remote command and exit with its exit status. The connection authenticates with the SSH agent.

Each argument is one word of the remote argument vector. The tool quotes every word, so the remote shell splits nothing, expands nothing, and globs nothing. Write -- before a word that starts with a hyphen. A caller that wants a remote shell asks for one:

$ fuguvm ssh -- sh -c 'cd /src && make regress'

The remote command gets no terminal and no standard input.

put [--mode octal] local remote

Copy the local file or directory local into the guest at the absolute path remote, over SFTP. remote is never a container: ‘put src /root’ writes the content of src into /root.

The tool writes each file to a temporary name beside its destination and then publishes every file with mv(1), so a failure leaves no partial file. A directory copy is not atomic as a whole. --mode sets the mode of each written file, as 3 or 4 octal digits. Without it, each file keeps its local permission bits masked with 0777. The tool refuses a symbolic link, a device node, a socket, a fifo, and a file above 64 MiB, before it writes one byte.

get remote local

Copy one regular guest file at the absolute path remote to the host file local. The tool reads the whole file, at most 64 MiB, and then writes local atomically with mode 0644. local must not be an existing directory.

console

Attach the terminal to the serial console of the guest with telnet(1). End the session with the telnet escape key, ‘Ctrl-]’, and then quit. The tool restores the terminal on every exit path. The console takes one client, so do not attach while an installation or an expect script runs.

expect script [argument ...]

Run an expect(1) script against the serial console. script is a path, or the name of a script under share/fuguvm/expect/. The script gets the host, the port, and each argument, in that order. A shipped script reads its timeout from FUGUVM_TIMEOUT. A script that fails exits with status 9.

cache list

List the image cache and the mirror cache. The command prints one line for each cached image, with its key, size, creation time, and snapshot count, and marks the entry that the invoked guest derives. It then prints the mirror downloads by OpenBSD version, and one ‘Distfiles’ line while the distfile tree holds a file.

cache clear [--stale]

Remove every cached image and every mirror download, and the temporary trees that an interrupted save left behind. With --stale, keep the image that the invoked guest derives and the downloads of the OpenBSD version that it installs. The flag also keeps the distfile tree and re-applies the distfile_cache cap.

The scope is one guest, so a removal for one guest removes the images of another guest with a different version or disk_size. The command refuses to remove an image that a running guest of the project uses, with status 5. It warns when a removal orphans the disk of a stopped guest.

snapshot save | restore | rm name

 

snapshot list [--names]

Manage the named snapshot layers of the cache entry that the invoked guest derives. See IMAGE CACHE.

save flattens the stopped working disk into the cache as a layer over its base image. The guest must be installed on a cached image. restore replaces the working disk with a new overlay on the layer, and reseeds the recorded state. It needs no disk and no state, so it runs on a fresh checkout before the first up. Both refuse a running guest with status 5. A layer that does not exist exits with status 11, so a script tells it from a failure:

$ fuguvm snapshot restore deps-$hash || provision_from_scratch

list prints one line for each layer on standard error. With --names, it prints the bare names on standard output.

image export path [--format qcow2 | raw]

Write the installed base image of the invoked guest as a full-disk image at path. The source is the base image of the cache entry that backs the working disk. A standalone disk exports the working disk itself. The default format is qcow2, and raw writes a sparse raw image. The command refuses a running guest with status 5, and an existing path with status 1. The write goes through one temporary sibling and one rename. The report on standard output follows the status contract, with the keys bytes, format, key, path and source.

mirror fetch file

Download one file of the release of the invoked guest, verify it under the release key, store it in the mirror cache, and print the cached path. The manifests decide the scope: the release scope when the SHA256 of the architecture directory names the file, and the source scope otherwise. See MIRROR CACHE.

mirror verify

Verify every cached file of the version and the architecture of the invoked guest against the two signed manifests. The command prints one line for each failed file and one summary line with the counts of ok, failed, and unknown files. It removes each failed file, and a cached manifest whose signature fails. A file that no proven manifest names is unknown, and the command keeps it. The command exits with status 1 when one file failed.

disk check | repair | info

Manage the working disk with qemu-img(1). check checks the integrity of the image. repair repairs it, and refuses a running guest. info prints the image report, in the key: value form of status.

help [command]

Print the usage of the tool, or of one command.

CONFIGURATION

fuguvm reads two files, both named .fuguvmrc: the global file in the home directory, and the project file at the project root. The project file wins. A setting is one name value or name = value line. A guest is a vm block, named by its guest name. A ‘#’ starts a comment. A line that does not parse is an error that names the file and the line, and the tool exits with status 3.

cache_dir  ~/.cache/fuguvm
default_vm default

vm "default" {
	arch         arm64
	version      7.8
	memory       2048
	disk_size    8G
	ssh_port     2222
	console_port 4444
}

The top-level settings are as follows:

The settings of a vm block are as follows. The first five also stand at the top level of either file, and the block wins.

Each path value expands a leading tilde and resolves a relative path against the project root. A distfile_cache line inside a block does not apply, and a qemu_version line there is an error. A yes/no value that is neither gives a warning and the default.

A guest resolves in this order: a vm block of the project file, a vm block of the global file, and the flat settings file .fuguvm/vms/name.conf. init writes the default guest as such a file.

INSTALLATION

The directives of a guest select one of three install modes, and a disk belongs to its mode and its architecture for its whole life. After a change of either, rebuild the guest with destroy and up.

Without autoinstall and base_disk, up installs with the shipped expect(1) script over the serial console. It generates a random root password, stores it in the state, and installs the ssh_pubkey over a password login. A failed script exits with status 9.

With autoinstall, up serves the response file to the guest from a loopback HTTP responder, and the file answers every installer question. The responder replaces each ‘@PROXY_URL@’ token in the file with the guest URL of the mirror proxy. The tool validates no answer in the file, and QEMU runs with -no-reboot, so the reboot at the end of the install cannot start a second one.

With base_disk, up installs nothing. It publishes the image as one cache entry for the whole project, once for each host, and overlays the working disk on it. The tool never writes to the source file. autoinstall and base_disk contradict each other, and base_disk needs image_cache yes.

Outside the default mode the tool generates no password. With ssh_pubkey and root_password_file, up reads the password from the file and installs the key over a password login. Without ssh_pubkey, it waits for SSH with the key of the operator, which the image must trust already. ssh_pubkey without root_password_file is a configuration error.

Before the first boot the tool proves the release manifest under the release key and verifies the miniroot image. The installer of a numbered release verifies its own sets. With verify yes the install script therefore stops on the “Continue without verification?” prompt with status 9, because the prompt reports a broken mirror. With verify no the tool downloads without a proof and the installer waives its check, with one warning on each side.

ARCHITECTURE

The arch directive selects the QEMU binary, the machine type, the firmware, the installer answers, and the image-cache key. up and start exit with status 3 when the QEMU binary of the architecture is not on PATH.

The accelerator follows the host. The tool selects KVM on a Linux host with a writable /dev/kvm, and HVF on a Darwin host, and only when the host machine runs the instruction set of the guest. It selects TCG in every other case, and under --emulate. An installation under TCG takes tens of minutes.

With qemu_version, up and start run the binary with --version before the first spawn and compare component by component, over the components that the directive names: 9.0 accepts 9.0.4 and refuses 9.1.0. A mismatch, an absent binary, and output with no version each exit with status 3.

ADDRESSES AND PORTS

QEMU binds the forwarded SSH port and the serial console to bind_address. The default, 127.0.0.1, lets the host alone reach a guest. The tool connects to the same address, or to 127.0.0.1 when the ports bind to 0.0.0.0.

With auto, up and start take the first free port of a fixed range of 100 ports above the default port of the directive, and record it. The probe skips every port that a sibling guest of the project holds or declares, running or not. An exhausted range exits with status 1. The record dies with the run: stop clears it, and a later start resolves again. Read the port with status.

ssh, put, get and console refuse a stopped guest with status 1. expect and wait refuse a stopped guest with a port of auto with the same status, because such a guest has no port.

IMAGE CACHE

An installed disk lands in the cache as a clean, compacted base image, and a later up overlays a throwaway working disk on it. With a warm cache, destroy and up are a cheap factory reset. The entries sit under cache_dir, beside the mirror cache:

<cache_dir>/proxy/...                               the mirror cache
<cache_dir>/installed/<key>/base.qcow2              mode 0400
<cache_dir>/installed/<key>/meta.json               mode 0600
<cache_dir>/installed/<key>/snapshots/<name>.qcow2  mode 0400
<cache_dir>/installed/<key>/snapshots/<name>.json   mode 0600

The key is version-arch-hash. The hash covers every input that shapes the installed disk of the mode: the version, the architecture, the mode, and the file share/fuguvm/cache-generation. The default mode adds disk_size and the content of install.exp. The autoinstall mode adds disk_size, the content of autoinstall.exp, and the content of the response file. The import mode adds nothing, so every guest of a project derives one key for one base_disk. The memory and port settings do not shape the disk, and a change to them hits the same entry.

The generation file is a counter. Increase the number when the install procedure changes outside the scripts, and keep the bare number in the file.

An entry is immutable and write-once. The tool builds it whole in a temporary directory and publishes it with one rename. A snapshot is an immutable named layer over the base image of an entry, and snapshot save flattens the working disk onto the base:

base.qcow2  <-  <name>.qcow2  <-  disk.qcow2

No snapshot is the parent of another, so the removal of one orphans none. Each operation that invalidates a base also invalidates its snapshots.

MIRROR CACHE

A proxy on the host caches what an installation and a ports build fetch from https://cdn.openbsd.org: the file sets, the packages, the syspatch sets, the kernels, the miniroot images, the manifests, and the four source tarballs of a release. With a distfile_cache cap above zero, it also caches the distfile tree under /pub/OpenBSD/distfiles/ of any mirror host. The proxy admits nothing else, and it caches no https body. The guest reaches it at the proxy_url of status, and the port changes between runs.

Every tree except the distfiles carries a version in its path, and cache clear --stale prunes the other versions. The distfile_cache cap bounds the distfile tree instead: the proxy evicts the oldest file first after each store.

The host verifies each download under the base key of the release with signify(1) before the file enters the cache. The version names the key file: 7.8 gives openbsd-78-base.pub. The tool reads the key from signify_dir, then from /etc/signify/, and then from the shipped share/fuguvm/signify/ tree, which holds the key of the default release. A guest of an older release needs one of the first two. The tool serves no snapshot release. Each scope holds one SHA256 manifest with one signature: the architecture directory signs the sets, the kernels and the miniroot, and the version directory signs the source tarballs.

A ports build in the guest needs two lines of guest configuration. http_proxy must name the proxy_url value, because ftp(1) in the guest reads it. The ports tree must fetch each distfile over http from the OpenBSD distfile mirror, because an https site never reaches the cache. SITE_BACKUP in bsd.port.mk(5) is that route:

$ fuguvm ssh -- sh -c \
	"echo 'SITE_BACKUP = http://ftp.openbsd.org/pub/OpenBSD/distfiles/' \
	>> /etc/mk.conf"
$ proxy="$(fuguvm status proxy_url)"
$ fuguvm ssh -- sh -c \
	"cd /usr/ports/devel/gmake && env http_proxy=$proxy make fetch"

The host verified the ports tree, and ‘make checksum’ verifies each distfile against its distinfo digest in the guest.

FILES

~/.fuguvmrc

The global configuration.

.fuguvmrc

The project configuration, at the project root.

.fuguvm/vms/<name>.conf

A guest declaration outside the two configuration files.

.fuguvm/state/<name>/

The state of one guest: the working disk, the JSON state, the PID files, and the logs.

~/.cache/fuguvm/

The default cache directory.

share/fuguvm/expect/

The shipped expect(1) scripts.

share/fuguvm/cache-generation

The generation counter of the image cache.

share/fuguvm/signify/

The signify public key of the default release.

EXIT STATUS

fuguvm exits with one of the following values:

ssh with a command exits with the status of the remote command. A remote status can read like a tool status, and the diagnostic on standard error tells the two apart.

EXAMPLES

Initialize a project, boot the default guest, and run a command in it:

$ fuguvm init
$ fuguvm up
$ fuguvm ssh -- uname -a
OpenBSD openbsd-test.my.domain 7.8 GENERIC arm64

Copy a source tree in, build it, and copy the report back:

$ fuguvm put src /root/src
$ fuguvm ssh -- sh -c 'cd /root/src && make regress > report.txt'
$ fuguvm get /root/src/report.txt report.txt

Declare an amd64 guest and a fleet of two guests with automatic ports in .fuguvmrc:

vm "openbsd-amd64" {
	arch      amd64
	disk_size 16G
}
vm "scenario-1" {
	ssh_port     auto
	console_port auto
}
vm "scenario-2" {
	ssh_port     auto
	console_port auto
}

$ fuguvm --vm openbsd-amd64 up
$ fuguvm --vm scenario-1 up
$ fuguvm --vm scenario-1 status ssh_port
2222

Cache a provisioned state, and restore it on the next checkout:

$ fuguvm ssh -- pkg_add gmake
$ fuguvm down
$ fuguvm snapshot save deps
$ fuguvm snapshot restore deps && fuguvm up

SEE ALSO

expect(1), qemu-img(1), signify(1), ssh(1), telnet(1), autoinstall(8)

The module documentation is in the .pod sidecars beside lib/App/FuguVM/.

HISTORY

The fuguvm utility was first available in 2025, to run the test suite of a daemon against a real OpenBSD guest.

AUTHORS

Dick Olsson <hi@senzilla.io>

SECURITY CONSIDERATIONS

The guest permits root login with a password. In the default install mode the tool generates that password and stores it in the state directory and in the meta.json of the cache entry, both with mode 0600. The cache makes the life of the secret longer: it survives destroy and changes only with the cache key.

The console port carries no authentication of its own. With the default bind_address only the host reaches a guest. With 0.0.0.0 both ports listen on every host interface, so treat such a guest as open to everyone who reaches the host, and treat a shared cache_dir the same way.

An autoinstall(8) response file and a root_password_file each hold a root password. The responder binds the loopback address only, and the tool never copies either file into the cache or the state directory. The cache key holds a digest of the response file, not its content. The tool warns when the group or others can read the password file.

The shipped key under share/fuguvm/signify/ is a trust anchor. A human adds each key file, and the commit that adds one states the comparison against /etc/signify/ on a real OpenBSD host of that release. verify no removes the host proof and the installer proof together. Use it only against a mirror that something else verifies.