The Guest Rootfs

The custom initramfs, the overlayfs root, and the initramfs-style switch_root that emberd-init performs as PID 1.

When a sandbox boots, the kernel runs emberd's own initramfs, a tiny cpio archive whose /init is a statically-linked emberd-init. As PID 1, that binary builds the real root filesystem and then serves the control plane.

The initramfs

rootfs/build.sh produces ~/firecracker-verify/emberd-initramfs.cpio:

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build -trimpath -ldflags="-s -w" -o "$STAGE/init" ./cmd/emberd-init
mkdir -p "$STAGE"/{proc,sys,dev,newroot,tmp}   # mountpoints emberd-init needs
( cd "$STAGE" && find . | cpio --create --format=newc ) > emberd-initramfs.cpio

Building with CGO_ENABLED=0 means the binary is fully static, so the initramfs carries no libc, no shell, nothing but /init and a handful of empty mountpoints (~2.5 MiB total). Everything else comes from the language pack.

The overlay root

The architecture wants a writable root without mutating the immutable base image, so emberd-init assembles an overlayfs:

lower: the language-pack squashfs, mounted read-only.
upper + work: a fresh tmpfs (per-VM, lives in RAM).
merged: the overlay of the two; this becomes the new root.

Writes land in the tmpfs upper; the squashfs is never touched and its pages are shareable across sandboxes. Reset semantics are trivial: the tmpfs dies with the VM.

The read-only squashfs (/dev/vda, lower layer) and a per-VM tmpfs (upper + work) are merged by overlayfs; switch_root makes the merged view the writable, per-VM, ephemeral root /.

The boot sequence

bootstrapPID1() runs only when os.Getpid() == 1 (so host-side unit tests of the same binary skip it):

Mount proc on /proc, devtmpfs on /dev (this is what makes /dev/vda appear), and sysfs on /sys, all inside the initramfs.
Mount the squashfs (/dev/vda) read-only at /lower.
Mount a tmpfs at /overlay; create upper/ and work/ on it.
mount -t overlay with lowerdir=/lower,upperdir=/overlay/upper,workdir=/overlay/work at /newroot.
MS_MOVE /proc, /dev, /sys into /newroot (carrying their subtrees).
switch_root into /newroot.
Set a default PATH and HOME (PID 1 starts with an empty environment, so python3 isn't resolvable until this is done).
Read emberd.interpreter= from /proc/cmdline (see language packs).
Start the child reaper (a SIGCHLD handler). As PID 1, emberd-init inherits any process a workload double-forks; the reaper wait4s them so they don't leak as zombies, while still letting runExec collect the interpreter's own exit code.
Serve the vsock control plane.

Why switch_root and not pivot_root?

pivot_root(2) is rejected on an initramfs (EINVAL). So emberd uses the busybox switch_root technique instead:

syscall.Chdir(newRoot)            // cd /newroot
syscall.Mount(".", "/", "", syscall.MS_MOVE, "")  // move merged root onto /
syscall.Chroot(".")               // chroot into it
syscall.Chdir("/")

Moving the /proc, /dev, /sys submounts into /newroot before the final MS_MOVE means they ride along when the whole subtree relocates to /.

You can confirm it worked from inside a running sandbox; /proc/mounts shows:

overlay / overlay rw,relatime,lowerdir=/lower,upperdir=/overlay/upper,workdir=/overlay/work 0 0

/ is the overlay, and writes anywhere (e.g. /root, /etc via copy-up) succeed.

Kernel requirements

The guest kernel needs SQUASHFS, OVERLAY_FS, DEVTMPFS, TMPFS, and VIRTIO_BLK (plus the vsock options from the control-plane page). The Firecracker CI kernel has all of them built in, verified by extracting its embedded config, which is why emberd boots on the stock kernel with no rebuild.

The kernel boot args still include panic=1, so if emberd-init ever exits or the bootstrap fails, the guest panics and the VM dies, which is the correct outcome for a sandbox whose agent is gone. The VM stays alive purely because PID 1 (emberd-init) blocks forever in its vsock accept loop; nothing on the host needs to hold the VMM's stdin open (it's wired to /dev/null).