Blocks to Sandboxes
How disks, filesystems, mounts and sandboxes fit together, built up one layer at a time.
A parent sandbox fans work out to many short-lived child sandboxes. Each child writes results the parent needs to read. The goal:
- Own storage per childIt outlives the child.
- 32+ at onceThe parent reads many at the same time.
- Picked after bootThe parent doesn't know which ones up front.
- Changes while runningThe set keeps changing.
Each layer is a short animation. Use Next, or click a figure and press ← →.
A drive is a row of numbered boxes
Every storage device speaks the same tiny language: numbered blocks of bytes.
A drive is a row of numbered blocks, 4 KB each. It has no idea what a file is.
You can ask it two things. One: write these bytes to block 5.
$ dd if=hi.bin of=/dev/vdb bs=4096 seek=5 count=1Two: read block 5. That's the whole interface.
$ dd if=/dev/vdb bs=4096 skip=5 count=1Linux calls this a block device and names it under
/dev. A VM's disks are/dev/vda,/dev/vdb, and so on.$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 253:0 0 2G 0 disk / vdb 253:16 0 10G 0 disk
A filesystem is an index on top of the blocks
A file is some blocks of data plus an entry in a table that maps its name to those blocks.
Blocks have numbers, not names. Something has to remember where
notes.txtlives.Reserve a few blocks for an index. Writing an empty index onto a drive is formatting.
$ mkfs.ext4 /dev/vdbSaving
notes.txt(8 KB) writes its bytes into two free blocks, 6 and 7.The index records it:
notes.txt→ blocks 6, 7.$ debugfs -R "stat /notes.txt" /dev/vdb | tail -2 EXTENTS: (0-1):6-7
Reading reverses it: look up the name, then read those blocks.
$ cat notes.txtThe index is the filesystem. Every layer above this one is about never letting two writers disagree about it.
Mounting attaches a disk to the folder tree
Programs open files by path, so a disk's files need a place in the tree.
Linux has one folder tree, rooted at
/. Every path lives in it.$ ls / home mnt tmp
A new disk,
/dev/vdb, has files on it, but nothing in the tree leads to them.Pick an empty folder to act as the doorway: the mount point.
$ mkdir -p /mnt/dataMounting adds a row to the kernel's mount table. The disk's files appear under the doorway.
$ mount /dev/vdb /mnt/data $ findmnt /mnt/data TARGET SOURCE FSTYPE OPTIONS /mnt/data /dev/vdb ext4 rw,relatime
To open
/mnt/data/notes.txt, the kernel walks the path, hits the mount, and continues in vdb's index.$ cat /mnt/data/notes.txtUnmounting deletes the row. The disk is untouched and can be mounted somewhere else.
$ umount /mnt/data
A mount is a row in a table: this folder leads to that filesystem. Mounting something into a sandbox means adding a row to the table of whichever kernel the sandbox runs on.
One disk, two kernels: why sharing breaks
Attach one ordinary disk to two machines at once and it gets corrupted, however careful each one is.
The kernel keeps a copy of the index in RAM and writes changes back later. This is the page cache.
Attach the same disk to machine B. B loads its own copy. Nothing keeps the two in sync.
A saves
a.txt. Its copy says block 9 is free, so A takes it.B saves
b.txt. B's copy is stale, so it takes block 9 too.Both write back. Block 9 holds b.txt's bytes, and
a.txtpoints at the wrong data. The disk is corrupted.Read-only doesn't fix it. A reader's cache goes stale and can catch blocks mid-write. ext4 and xfs assume one kernel per disk. Cluster filesystems like GFS2 and OCFS2 lift that rule with a lock manager, at a real cost.
Clouds enforce this. An EBS volume or a Persistent Disk attaches read-write to one machine at a time, unless you opt into multi-attach and bring a cluster filesystem.
A sandbox controls what a program can see
There are two common shapes. They differ in where the wall sits.
Programs can't touch hardware. They ask the kernel for everything, from opening a file to starting a process. Each request is a syscall.
A container is a normal process with a private view. Its mount namespace gives it its own tree and mount table, and its
/is a folder on the host.$ unshare --mount --pid --fork --root=/srv/sb1 /bin/shCgroups cap its CPU and memory. Seccomp blocks risky syscalls.
$ echo 4G > /sys/fs/cgroup/sb1/memory.maxMany containers share one kernel. Each sees only its own tree and processes.
That shared kernel is the weak spot. One bug in any of its hundreds of syscalls can expose every neighbor.
A microVM gives each sandbox its own kernel. The hypervisor, Firecracker here, fakes a small computer with a CPU, memory, a network card and disks.
The disk is a plain file on the host,
child-7.img, which the guest sees as/dev/vda. Firecracker is told about it before boot.$ curl --unix-socket fc.sock -X PUT localhost/drives/rootfs \ -d '{"drive_id":"rootfs","path_on_host":"child-7.img","is_root_device":true,"is_read_only":false}'The guest writes block 5, so Firecracker writes 4,096 bytes at offset 20,480 in
child-7.img. Layers 0 to 2 still happen, inside the guest.Escaping means breaking the guest kernel and then the hypervisor. AWS Lambda, Fly.io and E2B use Firecracker. Modal uses gVisor, a kernel that runs as an ordinary process.
In a container, adding a disk means adding a mount-table row on a shared kernel. In a microVM, it means plugging a fake device into a computer that is already running, which is much harder.
Adding a disk to a running sandbox
Sometimes you only know which disks you need after a sandbox has booted. Adding a block device to a running VM is called hotplug.
Before boot, the host fixes the VM's device list. The guest kernel scans it once while starting up.
With hotplug, the host adds a disk to the running VM. QEMU, Cloud Hypervisor and EC2 volumes support this.
/dev/vdcappears and the parent mounts it.$ mount -o ro /dev/vdc /mnt/child-7Slots run out, and each disk needs a ritual: stop the child, detach, attach, mount, read, unmount, detach. The child must finish first, because of layer 3.
Firecracker has no hotplug. Its device list is frozen at boot, so you can't pick disks later.
$ curl --unix-socket fc.sock -X PUT localhost/drives/child8 -d '{…}' The requested operation is not supported after starting the microVM.
Shortcuts that skip the device dance
Containers: with mount propagation, anything the host mounts into a shared folder later appears inside the running container. Kubernetes calls this mountPropagation: HostToContainer.
QEMU and Cloud Hypervisor VMs: virtiofs shares a host folder into the guest, and the host can add to it at any time. Firecracker doesn't support virtiofs. Both shortcuts need everything on one host.
Network filesystems put one server in charge
Keep exactly one kernel on the disk, and let everyone else talk to it over the network.
One machine, the file server, owns the disk and the only real index. That satisfies layer 3.
Sandboxes mount the server over the network. Inside,
/runs/42is just a folder.$ mount -t nfs files:/runs/42 /runs/42Child 2 writes
/runs/42/child-2/out.json. The request goes to the server, which does the block work.Child 2 closes the file. NFS guarantees close-to-open consistency: whoever opens it next sees all of it.
The parent reads the file from the server's single index. There are no stale copies.
A network mount is just a connection. It works any time after boot, needs no device slot, and serves thousands of clients. The sandbox must allow NFS or FUSE mounts.
$ mount-s3 my-bucket /runs # FUSE over S3, one of several options
Managed NFS: AWS EFS, Google Filestore. Object storage presented as a folder through FUSE: JuiceFS, mountpoint-s3, gcsfuse.
Mount one place. Let the folders change.
Back to the running example. Give each child a folder inside one shared mount instead of its own disk, and every requirement becomes easy.
The parent boots with one mount,
/runs/42, and it never changes.N = 8 tasks, at most X = 3 running. Each child owns one folder and is its only writer.
Child 1 writes
out.json.tmp, then renames it toout.json. Renames are atomic, so the parent never reads half a file. S3 uploads are already atomic.$ produce_results > /runs/42/child-1/out.json.tmp $ mv /runs/42/child-1/out.json.tmp /runs/42/child-1/out.json
Child 1 exits and its sandbox is destroyed. Its folder stays. Task 4 takes the free slot.
Fast-forward until the queue drains. The parent's mounts never changed.
The parent reads every result whenever it likes, through the same single mount for 8 children or 8,000. A lifecycle rule can expire old runs.
$ ls /runs/42/*/out.json | wc -l 8
Mounts are rigid and folders are cheap. Mount one shared place at boot and let folders come and go.
Four ways to build it
The columns are the four requirements from the top of the page.
| Approach | Own storage per child | 32+ at once | Picked after boot | Changes while running |
|---|---|---|---|---|
| Block disk per child, attached to the parentOne kernel per disk, so each child must finish first. Impossible on Firecracker. | yes | slots run out | hotplug only | hotplug only |
| Host folder shared into the parentMount propagation or virtiofs. Needs provider support and one host. | yes | yes | yes | yes |
| Network filesystem, a folder per childNFS, EFS, JuiceFS. The sandbox must allow NFS or FUSE mounts. | yes | yes | yes | yes |
| Object storage, a prefix per childS3, R2, GCS. No mount at all; code uses an API instead of plain files. | yes | yes | yes | yes |
Glossary
- block
- A fixed-size numbered box on a drive, usually 4 KB.
- block device
- Anything that reads and writes numbered blocks, like
/dev/vda. - filesystem
- The index that maps names to blocks. ext4 and xfs are examples.
- formatting
- Writing an empty index onto a drive with
mkfs. - mount point
- The empty folder where a filesystem gets attached.
- mount table
- The kernel's list of folder-to-filesystem rows.
- page cache
- The kernel's RAM copy of disk data and index.
- namespace
- A private view the kernel gives a process: its own mounts, processes or network.
- cgroup
- A CPU and memory limit on a group of processes.
- hypervisor
- Software that fakes hardware so a guest kernel can run. Firecracker is one.
- virtio-blk
- The fake disk device a VM guest talks to.
- hotplug
- Adding a device to a VM that is already running.
- FUSE
- Lets an ordinary program act as a filesystem.
- close-to-open
- NFS's promise that a closed file is fully visible to the next opener.