HW2: A VT-x hypervisor for xv6 (part 2, “Booting xv6”)

This is the second homework in the series in which you build a hypervisor for xv6-64 that runs inside xv6 itself. In HW1 your hypervisor ran a small “hello world” guest. In this homework it runs a complete copy of xv6 as the guest: the guest boots, mounts its own file system from an emulated disk, and gives you an interactive shell, all inside a VM that your xv6 kernel created.

$ runxv6 xv6kernel xv6fs.img
runxv6: booting xv6kernel (type Ctrl-] to stop)
xv6...
cpu0: starting 0
cpu0: no VMX support
sb: size 2000 nblocks 1941 ninodes 200 nlog 30 logstart 2 inodestart 32 bmap start 58
init: starting sh
$ ls
.              1 1 512
..             1 1 512
README         2 2 1688
cat            2 3 17736
...

Everything after the first line is printed by the guest. cpu0: no VMX support is the guest’s own vmxinit() running, and finding that your hypervisor hides VT-x from it.

HW1 was about the CPU side of virtualization: VMX operation, the VMCS, the EPT, entering and leaving the guest. HW2 is about everything a real operating system needs besides a CPU:

• memory it can be given and read back, including addresses that only make sense through the guest’s own page tables;
• firmware: the guest kernel must be loaded somewhere and must find the tables a BIOS would have left for it;
• interrupts: a timer, and a way for devices to say “I am done”;
• devices: a console to type into and a disk to boot from.

As in HW1, the work is split between the kernel (vmx.c), which does what only the kernel can do, and a user program, now runxv6, which plays the role QEMU plays on top of KVM: it pretends to be the PC around the guest.

How to use this handout

The handout has the same shape as HW1’s. Each part has an introduction, a table of what is given and what you write, steps with code to read or paste (each with the sections of the Intel manual it relies on), a few Your turn steps where you write the code yourself, a checkpoint, and questions whose answers go in answers-hw2.md.

Two things are new:

• You build on your own HW1 code. Some HW2 steps add a few lines inside functions you wrote or edited in HW1 (vmcsinit, vmcreate, vmxhandle, vmrun). Those steps show the exact lines and where they go. The new functions and the new program runxv6.c come with the starter code.
• The checkpoints build on each other more. The guest needs its interrupts and its disk before it gets anywhere interesting, so the first time you see the full boot is the end of Part 5. Earlier parts tell you what partial output to expect.

The section numbers are those of the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026), written “SDM”, as in HW1. This edition renamed two fields you will use in Part 4: the “IDT-vectoring information” is now the “original-event identification”, and the “VM-entry interruption information” is now the “injected-event identification”. The handout gives both names.

Questions that came up while this homework was being built are answered where they arise, often under a heading or in bold phrased as the question: “Where is the guest kernel loaded?” (Part 3), “Why the highest vector first?” (Part 4), “How does the guest’s keyboard work?” and “Are we ignoring the guest’s screen?” (Part 5), and so on.

Getting the code

HW2 continues your HW1 repository. The starter code for HW2 is the hw2-assign branch of the course repository; merge it into your HW1 work:

$ cd xv6-vmx-homeworks            # your HW1 clone
$ git commit -am "HW1 final"      # if you have uncommitted work
$ git fetch origin
$ git merge origin/hw2-assign

hw2-assign adds new files (runxv6.c, timertest.c) and new functions, and changes given files (the file system, the console, the Makefile, the headers). It does not change the functions you wrote in HW1, so the merge should go through without conflicts. If git reports a conflict, keep your own version of the function and apply the HW2 step for it from this handout by hand.

Your hypervisor from HW1 must work: run runvm helloguest first. If you did not finish HW1, ask the course staff before you start.

Where to run it

Exactly as in HW1: QEMU with KVM on the CADE lab machines, or Bochs anywhere else (see the HW1 handout for the setup). Under Bochs, everything runs inside an emulated CPU, so the guest xv6 is emulated twice over; it is still quick enough, a few seconds to boot.

The new commands:

$ runxv6 xv6kernel xv6fs.img      # boot the guest

In the guest, use the shell as usual. Type Ctrl-] to stop the guest and return to the host’s shell. (Ctrl-C does nothing in xv6, host or guest.)

xv6kernel is the guest kernel and xv6fs.img its disk image. Both are built by make fs.img and stored in the host’s file system; Part 1 explains how.

Debugging

The shared debugging guide applies unchanged: you debug the host kernel, with gdb attached to QEMU or with the Bochs debugger. The guest is just code that runs between vmx_enter and vmx_exit; to see what it is doing, print from runxv6 (every exit it sees) or from vmxhandle in the kernel. When the guest stops at an address, look it up in the guest kernel, not the host’s (they are built from the same source but are different binaries):

$ x86_64-elf-objdump -S xv6kernel > xv6kernel.asm     # then search for the address

The big picture

Two xv6s, one CPU

The guest kernel is the same xv6-64 source as the host, compiled a second time for a smaller machine (32 MB of memory, a 2000-block disk).

A run, step by step

1. Load. runxv6 reads xv6kernel, an ELF file, and copies its segments into an empty 32 MB VM with vmmem() (Part 3). There is no BIOS and no boot sector: xv6-64’s entry point expects 32-bit protected mode with paging off, which is exactly the state your HW1 vmcsinit already sets up.
2. Firmware tables. runxv6 writes the few bytes a BIOS would have left in memory, most importantly the MP tables that describe one CPU and one I/O APIC (Part 3).
3. Boot. The guest kernel runs. It switches to 64-bit mode (the kernel emulates its EFER write, as in HW1), sets up its page tables and its IDT, and starts talking to “hardware”. Every port I/O instruction exits to runxv6, which emulates the device (Part 5). CPUID exits to the kernel, which hides VT-x (Part 3).
4. Interrupts. The guest’s timer, its serial port and its disk need interrupts. runxv6 decides when one happens and asks the kernel to inject it; the kernel delivers it through the guest’s IDT at the next VM entry, as soon as the guest can take it (Part 4).
5. Disk. The guest reads its file system with rep insl, which moves 512 bytes straight into a buffer at a guest-virtual address. The kernel decodes the instruction, and runxv6 copies the data through the guest’s own page tables (Parts 2 and 5).
6. Console. What the guest prints arrives at runxv6 one character at a time through the serial port; what you type goes the other way, into the guest’s serial port, with an interrupt (Part 5).

What the guest’s PC has

The parts

Part 1: The host side: what changes in xv6 itself

Before your hypervisor can boot xv6, the xv6 it runs on has to change. Three things are missing from the HW1 tree:

• A guest kernel and a guest disk. runxv6 needs an xv6 kernel built for the small machine it emulates (32 MB of RAM, a 2000-block disk), and a disk image for it. The Makefile now builds a second kernel, xv6kernel, from the same sources, and a second file system image, xv6fs.img.
• Large files. Both of these have to be files in the host’s file system, so that runxv6 xv6kernel xv6fs.img can open them. Stock xv6 limits a file to 70 KB. The kernel is about 270 KB and the disk image 1 MB.
• A console that does not get in the way. runxv6 must keep the guest running and still pass every key you type to it. Stock xv6 cannot read the console without blocking, and it echoes and buffers what you type a line at a time.

All of this is given in the starter tree. Nothing in this part is yours to write: read the code, make sure you understand why it is there, build it, and answer the questions at the end. The later parts depend on it, and so does your debugging: when something goes wrong in Part 5, it helps to know what consraw() does to your terminal.

The part ends with a tour of the new system-call interface in vmxapi.h, which Parts 2 to 5 fill in behind.

What is given and what you write

You write nothing in this part.

Where to look things up. Most of this part is ordinary xv6. The xv6 book’s file system chapter (the inode layer and its “inode content” section) describes the stock block map that Steps 4 to 7 change. Step 13 cites the SDM, as before: the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026).

Step 1: Two kernels from one source: the guest’s parameters

Reference: Makefile (the “xv6 guest” section), memlayout.h, param.h.

The guest is xv6 itself, compiled a second time. Almost nothing in the kernel has to differ between the host and the guest, because your hypervisor makes the guest’s machine look like a PC. Two numbers do differ: how much physical memory the machine has, and how big its disk is. The Makefile names them once:

# Size of the guest's disk in blocks. The guest kernel (ide.c) and
# mkfsguest are both compiled with it. 2000 leaves room to run stressfs
# and usertests2 in the guest; 1000 started almost full.
GUESTFSSIZE = 2000
# PHYSTOP 0x2000000 is 32 MB, the guest memory runxv6 asks vmcreate()
# for (GUESTMEM in runxv6.c); the two must agree. memlayout.h and
# param.h use these values instead of their defaults.
GUESTDEFS = -DPHYSTOP=0x2000000 -DFSSIZE=$(GUESTFSSIZE)

and the two headers let a -D on the compiler’s command line win over their defaults:

// HW2: xv6 assumes it has physical memory up to PHYSTOP (224 MB here)
// and hands all of it to kalloc(). A guest has only what runxv6 gives
// it (GUESTMEM, 32 MB), and the EPT maps nothing above that, so the
// guest kernel is built with -DPHYSTOP=0x2000000 (Makefile GUESTDEFS);
// otherwise its kinit would touch unmapped guest-physical memory and
// cause an EPT violation. The #ifndef lets the -D value win.
#ifndef PHYSTOP                     // the guest kernel is built smaller
#define PHYSTOP 0xE000000           // Top physical memory
#endif

// HW2: FSSIZE is the size of the disk that holds the root file system.
// mkfs builds an image of exactly FSSIZE blocks, and ide.c panics on a
// block number at or past it. The host disk grew from stock xv6's 1000
// blocks to 4000, so that it can also hold xv6kernel and xv6fs.img
// (each 1 MB or less); dot-bochsrc gives Bochs a disk of the same size.
// The #ifndef lets the Makefile build the guest kernel and mkfsguest
// with -DFSSIZE=$(GUESTFSSIZE), for the guest's smaller disk; both must
// agree with the image runxv6 serves.
#ifndef FSSIZE                      // the guest file system is smaller
#define FSSIZE       4000  // size of file system in blocks
#endif

PHYSTOP: why the guest kernel must know its memory size. xv6 does not probe for memory. It assumes RAM from 0 up to PHYSTOP, and at boot kinit1() and kinit2() put every page below PHYSTOP on the free list. kfree() fills each page with junk (memset(v, 1, PGSIZE)) as it does so, so the kernel writes to every one of those pages early in boot, before it starts its first process. The guest’s “RAM” is exactly what runxv6 asked vmcreate() for:

#define GUESTMEM (32*1024*1024)

Your EPT maps guest-physical 0 to GUESTMEM and nothing above it except the two APIC pages. If the guest kernel kept the host’s PHYSTOP (224 MB), its kinit1() would walk up through memory and, at the first page past 32 MB, touch a guest-physical address the EPT does not map. The CPU exits with an EPT violation, and runxv6 stops with guest touched unmapped guest-physical 0x2000000. The opposite mistake (a PHYSTOP smaller than GUESTMEM) is harmless: the guest simply never uses the rest. The two numbers live in different files (Makefile and runxv6.c), and nothing checks that they agree, so if you change one, change the other.

Why 32 MB? It has to be at most VM_MAXMEM (64 MB), or vmcreate() refuses it. Every guest page is a host page from kalloc(), so 32 MB is 8192 of the host’s pages, out of the host’s 224 MB; you can still run other programs next to the guest. xv6’s programs are tiny, so 32 MB is plenty for them. (The one exception is usertests1, whose sbrktest wants 100 MB; it fails in the guest, as expected.) And memory costs time: vmcreate() zeroes every page and the guest’s kinit writes every page again, which you notice under Bochs.

FSSIZE: why the guest kernel must know its disk size. FSSIZE is used in two places. mkfs builds an image of exactly FSSIZE blocks, and the kernel’s IDE driver refuses any larger block number:

  if(b->blockno >= FSSIZE)
    panic("incorrect blockno");

So the guest kernel, mkfsguest, and the image runxv6 serves must all agree on 2000.

Why 2000 blocks? The guest’s disk holds the same programs as the host’s, minus two (Step 3), so it starts with about 820 blocks in use. The first version used stock xv6’s 1000 blocks, and the disk started with fewer than 200 free blocks: running stressfs and then usertests2 in the guest ran out of blocks. With 2000 blocks the guest prints sb: size 2000 nblocks 1941 ... at boot, and has room for the tests. The host’s disk grew too, from 1000 to 4000 blocks, because it now holds both guest files (Step 3); dot-bochsrc and dot-bochsrc-nox describe a 4000-sector disk to match.

What is stressfs? One of xv6’s test programs: it forks five processes, each of which writes a 10 KB file and reads it back, all at the same time. Nothing runs it unless you type it (or the autograder does). In the guest it is a good test of the disk emulation under concurrency: several processes wait for the disk at once, and every request must get its interrupt (Parts 4 and 5).

Step 2: Two kernels from one source: the guest/ directory

Reference: Makefile; the GNU make manual on pattern rules.

The guest kernel is built from the same .c and .S files as the host kernel, but with GUESTDEFS. Its object files go in a directory of their own:

# The guest kernel's objects go in their own directory, guest/. They
# come from the same sources as the host kernel's but are compiled with
# other -D values, so they must not share file names: in one directory,
# building one kernel would overwrite the other's bio.o and so on, and
# make, which judges only by timestamps, would link a kernel from
# objects built for the other one.
GUESTOBJS = $(addprefix guest/,$(OBJS))
# The guest's programs: the host's, minus the two VT-x monitors. The
# guest has no VMX (vmxcpuid() hides it), so they could only fail.
GUESTUPROGS = $(filter-out _runvm _runxv6,$(UPROGS))

# The guest kernel's C files. "$(OBJS): CFLAGS += ..." above applies
# only to the targets in OBJS (bio.o, ...), not to guest/bio.o, so the
# kernel flags are repeated here. CFLAGS has -MD, so gcc also writes
# guest/%.d, the headers each object depends on. make does not create
# directories itself, hence mkdir -p (@ keeps make from echoing it).
guest/%.o: %.c
	@mkdir -p guest
	$(CC) $(CFLAGS) -m64 -mcmodel=large -DX86_64 $(GUESTDEFS) -c -o $@ $<

# The guest kernel's assembly files (entry.S includes memlayout.h and
# param.h), with the same -D values as the C files.
guest/%.o: %.S
	@mkdir -p guest
	$(CC) $(ASFLAGS) $(GUESTDEFS) -c -o $@ $<

Why not build the guest kernel in the same directory, say with make kernel GUEST=1? Because make decides what to rebuild only by comparing timestamps. kalloc.o built for the host and kalloc.o built for the guest have the same name and the same sources, so make cannot tell them apart: after building one kernel, it would consider the other one’s objects up to date, and quietly link a guest kernel with the host’s PHYSTOP (or the reverse). Separate file names (guest/kalloc.o) make the two builds independent, and both kernels are always correct after a plain make.

Two details. -include guest/*.d at the end of the Makefile reads the header dependencies gcc -MD writes, so editing param.h rebuilds the guest objects too. And entry.S is compiled twice as well, since it includes memlayout.h.

Step 3: xv6kernel, mkfsguest, xv6fs.img and the new fs.img

Reference: Makefile.

# Linked like kernel, with the same linker script, from the guest
# objects. initcode and entryother come from the host build: they do
# not use PHYSTOP or FSSIZE, so the same binaries serve both kernels.
# runxv6 loads the ELF file itself, so no boot block or xv6.img-style
# disk is needed.
xv6kernel: $(GUESTOBJS) guest/entry.o entryother initcode kernel.ld
	$(LD) $(LDFLAGS) -T kernel.ld -o xv6kernel guest/entry.o $(GUESTOBJS) -b binary initcode entryother

# mkfs for the guest's disk. mkfs takes the image size from FSSIZE at
# compile time, so the guest's needs its own binary, built with
# GUESTFSSIZE. It depends on the Makefile because GUESTFSSIZE is set
# here: changing it must rebuild mkfsguest.
mkfsguest: mkfs.c fs.h param.h Makefile
	gcc -Werror -Wall -DFSSIZE=$(GUESTFSSIZE) -o mkfsguest mkfs.c

# The guest's disk image: GUESTFSSIZE blocks with the guest's programs.
xv6fs.img: mkfsguest README $(GUESTUPROGS)
	./mkfsguest xv6fs.img README $(GUESTUPROGS)

# The host's disk. HW2 adds xv6kernel and xv6fs.img as ordinary files,
# so that, inside xv6, "runxv6 xv6kernel xv6fs.img" can read them.
# Both are larger than stock xv6's 70 KB file limit, hence the
# doubly-indirect block in fs.h.
fs.img: mkfs README $(UPROGS) helloguest xv6kernel xv6fs.img
	./mkfs fs.img README $(UPROGS) helloguest xv6kernel xv6fs.img

Four products, nested like Russian dolls:

• xv6kernel is an ELF file, linked exactly like kernel but from guest/*.o. There is no boot block: in HW2, runxv6 plays the boot loader and copies the ELF segments into guest memory itself (Part 3). initcode and entryother are shared with the host build, since they do not depend on PHYSTOP or FSSIZE.
• mkfsguest is the host-side (Linux or macOS) mkfs program, compiled with -DFSSIZE=2000. mkfs learns the image size at compile time from param.h, so a different size needs a different binary.
• xv6fs.img is the guest’s disk: 2000 blocks (1,024,000 bytes) holding README and the guest’s programs.
• fs.img is the host’s disk, as always, now 4000 blocks. Besides the host’s programs and helloguest, it holds xv6kernel and xv6fs.img as two ordinary files. Inside the host xv6, runxv6 opens them with open() and reads them with read(), like any other file, and keeps the disk image in memory while the guest runs: the guest’s writes change only that copy, never xv6fs.img.

Why the guest gets no runvm or runxv6. GUESTUPROGS filters them out. The guest’s CPU has no VMX: your vmxcpuid() (Part 3) clears the VMX bit, so the guest’s own vmxinit() prints cpu0: no VMX support, and every vmcreate() in the guest would fail. The two monitors could only print an error, and they would cost about 60 KB of a small disk. (Running a hypervisor inside the guest is nested virtualization: your hypervisor would have to emulate VMX instructions for the guest. That is far beyond this homework.)

timertest, new in UPROGS, is on both disks: it is a test for Part 4, meant to be run in the guest.

Step 4: Large files: the limit and the new inode layout

Reference: fs.h; the xv6 book, file system chapter (inode content); figures/fs-layout.svg.

A stock xv6 inode holds 13 block addresses. addrs[0] to addrs[11] point straight at data blocks; addrs[12] points at one indirect block, a block full of addresses. A block is 512 bytes and an address 4, so an indirect block lists 128 blocks. The largest file is therefore 12 + 128 = 140 blocks, or 71,680 bytes: 70 KB.

That is far too small. xv6kernel is about 270 KB (273,608 bytes in the reference build, 535 blocks) and xv6fs.img is 1,024,000 bytes (2000 blocks). Without a change, mkfs stops on its assert(fbn < MAXFILE) while building fs.img, and even if it did not, xv6 could not read past the 70 KB mark.

HW2 replaces the singly-indirect block with a doubly-indirect one:

// Block addresses in an inode (HW2): NDIRECT direct blocks, then one
// doubly-indirect block. That block lists NINDIRECT indirect blocks,
// each of which lists NINDIRECT data blocks, so a file can hold a whole
// guest kernel or disk image (up to about 8 MB). Stock xv6 had a
// singly-indirect block in the same slot, which limited a file to
// 12 + 128 blocks (70 KB). See figures/fs-layout.svg.
//
// NDIRECT stays 12, and addrs[NDIRECT] is still the only pointer to
// more blocks, so struct dinode keeps its 13 addresses and its 64
// bytes: 8 inodes still fit exactly in a 512-byte block (IPB below,
// and mkfs asserts that BSIZE is a multiple of the inode size), and the
// in-memory struct inode (file.h) is unchanged. The cost is that a file
// just past 12 blocks needs two pointer blocks (the doubly-indirect one
// and its first indirect one) instead of one.
#define NDIRECT 12
// Block addresses per indirect block: 512 / 4 = 128.
#define NINDIRECT (BSIZE / sizeof(uint))
// Data blocks reached through the doubly-indirect block: 128 * 128.
#define NDINDIRECT (NINDIRECT * NINDIRECT)
// Largest file in blocks: 12 + 16384 = 16396 blocks, 8,394,752 bytes.
#define MAXFILE (NDIRECT + NDINDIRECT)

The figure shows the whole path. The disk (top) is unchanged except for its size. The inode (middle) is byte for byte the stock one: 12 bytes of header and 13 four-byte addresses, 64 bytes, so 8 inodes still fill a 512-byte block exactly. Only the meaning of addrs[12] changed (bottom): it now points to a block of 128 addresses of indirect blocks, each of which lists 128 data blocks.

To find block bn of a file (counting from 0):

• if bn < 12, the address is addrs[bn];
• otherwise let d = bn - 12. Entry d / 128 of the doubly-indirect block names an indirect block, and entry d % 128 of that names the data block.

So file blocks 12 to 139 go through indirect block 0, blocks 140 to 267 through indirect block 1, and so on, up to 12 + 128 x 128 = 16,396 blocks (8,394,752 bytes, about 8 MB). On a 4000-block disk, no file can get anywhere near that.

Only three pieces of code know the shape of this tree: bmap() and itrunc() in the kernel, and iappend() in mkfs. The next three steps show them. readi(), writei() and everything above them call bmap() and never see the difference. file.h does not change, because struct inode keeps addrs[NDIRECT+1].

Step 5: bmap(): finding (and allocating) a block

Reference: fs.c.

bmap(ip, bn) returns the disk address of block bn of the file, allocating the block, and any pointer block on the way to it, if it does not exist yet. The direct case is unchanged. This is the new indirect case:

  bn -= NDIRECT;

  // From here on bn counts blocks past the direct ones, 0 to
  // NDINDIRECT-1. Write it as bn = i * NINDIRECT + j: i = bn / NINDIRECT
  // picks the indirect block (the entry in the doubly-indirect block),
  // and j = bn % NINDIRECT picks the data block within it. So blocks
  // 12-139 of the file go through indirect block 0, blocks 140-267
  // through indirect block 1, and so on.
  if(bn < NDINDIRECT){
    // Load the doubly-indirect block, allocating if necessary, then the
    // indirect block it lists for this bn, then the data block.
    // A new doubly-indirect block only changes the in-memory inode; the
    // caller (writei) writes the inode to disk with iupdate().
    //
    // A write here can log one more pointer block than in stock xv6
    // (two at a 128-block boundary). It still fits: filewrite() writes
    // at most 1536 bytes per transaction, so at worst 4 data blocks,
    // the inode, the bitmap, this block and 2 indirect blocks, 9 of the
    // MAXOPBLOCKS (10) a transaction may write.
    if((addr = ip->addrs[NDIRECT]) == 0)
      ip->addrs[NDIRECT] = addr = balloc(ip->dev);
    // balloc() zeroes new blocks, so a new pointer block lists nothing.
    bp = bread(ip->dev, addr);
    a = (uint*)bp->data;
    if((addr = a[bn / NINDIRECT]) == 0){
      a[bn / NINDIRECT] = addr = balloc(ip->dev);
      // The new pointer is only in the buffer cache. log_write() makes
      // it part of the current transaction: it is written to disk at
      // commit, together with the bitmap change in balloc(), and the
      // buffer stays cached until then. Without it, the pointer would
      // be lost once the buffer was reused, and the block would stay
      // allocated but unreachable.
      log_write(bp);
    }

The first level. If the file has no doubly-indirect block yet, allocate one. balloc() returns a zeroed block, so all 128 entries of a new pointer block are 0, which means “not allocated yet”. Then read it and look up entry bn / NINDIRECT; if it is 0, allocate that indirect block and record it with log_write().

log_write() is the step people forget. Changing a[...] changes only the copy in the buffer cache. Unless the buffer is logged, the change never reaches the disk: when the cache reuses the buffer, the new pointer is gone, and the block that balloc() marked as used is allocated but unreachable, a leak that itrunc() can never free. The new address in the inode (ip->addrs[NDIRECT]) needs no log_write() here, because writei() calls iupdate() after every write that touches the inode.

    // Release the doubly-indirect block before reading the next level:
    // nothing more is needed from it, and holding two buffers at once
    // would only use up more of the small buffer cache.
    brelse(bp);
    bp = bread(ip->dev, addr);
    a = (uint*)bp->data;
    if((addr = a[bn % NINDIRECT]) == 0){
      a[bn % NINDIRECT] = addr = balloc(ip->dev);
      // Same reason as above, for the pointer to the data block.
      log_write(bp);
    }
    brelse(bp);
    return addr;
  }

  panic("bmap: out of range");
}

The second level is the same pattern, one block further down: read the indirect block, look up entry bn % NINDIRECT, allocate the data block if needed, log the pointer block. The code releases the doubly-indirect buffer before reading the indirect one, so it never holds more than one buffer: xv6’s cache has only NBUF (30) buffers, shared by every process.

Does a write still fit in one log transaction? xv6 bounds every transaction by MAXOPBLOCKS (10) distinct blocks, and filewrite() splits large writes into pieces small enough to respect it. With one more level of pointer blocks, a piece can now dirty one more block than before, so the comment in the code checks the arithmetic: a 1536-byte piece touches at most 4 data blocks, plus the inode, the bitmap, the doubly-indirect block and (at a 128-block boundary) two indirect blocks: 9 blocks, still within 10.

Step 6: itrunc(): freeing the tree

Reference: fs.c.

When the last link to a file goes away and nothing holds it open, itrunc() frees all its blocks. The direct blocks are freed as before; the indirect part now walks two levels:

  // HW2: the doubly-indirect tree. Free it bottom up: the data blocks,
  // then the indirect block that listed them, and the doubly-indirect
  // block last. A pointer block is the only record of its children, so
  // it must stay allocated while it is being read: once freed, another
  // process's balloc() could take it and zero it before we got to its
  // entries, and those blocks would never be freed. A zero entry means
  // no block was allocated there; skip it.
  //
  // bfree() changes only the bitmap, and log_write() logs a block once
  // per transaction however often it changes, so truncating even an
  // 8 MB file logs just the bitmap and the inode: one transaction.
  if(ip->addrs[NDIRECT]){
    // Free each indirect block's data blocks, then the indirect block.
    bp = bread(ip->dev, ip->addrs[NDIRECT]);
    a = (uint*)bp->data;
    for(j = 0; j < NINDIRECT; j++){
      if(a[j]){
        // Indirect block j: read it, free the data blocks it lists,
        // and release its buffer before freeing the block itself.
        bp2 = bread(ip->dev, a[j]);
        a2 = (uint*)bp2->data;
        for(k = 0; k < NINDIRECT; k++)
          if(a2[k])
            bfree(ip->dev, a2[k]);
        brelse(bp2);
        bfree(ip->dev, a[j]);
      }
    }
    // Last, the doubly-indirect block. The pointer blocks themselves are
    // not cleared: once freed they are unreachable, and balloc() zeroes
    // a block when it hands it out again.
    brelse(bp);
    bfree(ip->dev, ip->addrs[NDIRECT]);
    ip->addrs[NDIRECT] = 0;
  }

Two rules make this correct.

• Bottom up. A pointer block is the only record of where its children are. If itrunc() freed the doubly-indirect block first, another process could allocate it (and balloc() would zero it) before the loop had read all its entries, and the blocks they named would leak. So the children are freed first, the parent last.
• Skip zeros. Files can have holes in the tree: a file of 20 blocks uses only entry 0 of the doubly-indirect block, and only entries 0 to 7 of that indirect block. A 0 entry means “never allocated”, and freeing block 0 would be a disaster, so the code checks every entry.

Here the code holds two buffers at once (bp and bp2), unlike bmap(): it needs the doubly-indirect block for the whole outer loop. Two of 30 buffers is fine.

Why does this not overflow the log? An 8 MB file has over 16,000 blocks. But bfree() writes only the bitmap block, and the log records each block once per transaction no matter how often it changes. With a 4000-block disk, the whole bitmap is one block. So the transaction is the bitmap plus the inode.

Step 7: mkfs: building files with the same tree

Reference: mkfs.c.

mkfs runs on your development machine, not in xv6. It writes the files given on its command line into a new image, so it must lay out large files exactly as bmap() expects to find them. Its iappend() gets the same two-level lookup:

      // Doubly indirect (HW2): addrs[NDIRECT] lists indirect blocks,
      // which list data blocks. The same tree as bmap() in fs.c, which
      // must be able to read what mkfs writes.
      //
      // dbn is the block number past the direct blocks; dbn / NINDIRECT
      // selects the indirect block, dbn % NINDIRECT the entry in it.
      uint dbn = fbn - NDIRECT;
      uint ind;  // block number of that indirect block, host byte order
      // mkfs allocates blocks in order from freeblock; main() marks
      // everything below freeblock in use in the bitmap at the end.
      if(xint(din.addrs[NDIRECT]) == 0){
        din.addrs[NDIRECT] = xint(freeblock++);
      }
      // Every sector was zeroed at the start, so a new pointer block
      // reads back as all zeros: no entries yet. Comparing with 0
      // needs no xint(), since 0 is 0 in either byte order.
      rsect(xint(din.addrs[NDIRECT]), (char*)indirect);
      if(indirect[dbn / NINDIRECT] == 0){
        indirect[dbn / NINDIRECT] = xint(freeblock++);
        wsect(xint(din.addrs[NDIRECT]), (char*)indirect);
      }
      ind = xint(indirect[dbn / NINDIRECT]);
      // Reuse the indirect[] buffer for the second level: the
      // doubly-indirect block is on disk already and is read again for
      // the next block of the file.
      rsect(ind, (char*)indirect);
      if(indirect[dbn % NINDIRECT] == 0){
        indirect[dbn % NINDIRECT] = xint(freeblock++);
        wsect(ind, (char*)indirect);
      }
      x = xint(indirect[dbn % NINDIRECT]);

The logic mirrors bmap(), with mkfs’s simpler allocator: blocks are handed out in order from freeblock, and the bitmap is written once at the end. xint() converts between the host’s byte order and xv6’s little-endian disk format; on an x86 or ARM machine it does nothing, but the code keeps it correct everywhere.

If bmap() and iappend() disagreed, for example if one used bn / NINDIRECT where the other used bn % NINDIRECT, mkfs would build a disk that the kernel reads as garbage past the first 6 KB of every large file, and nothing would complain until runxv6 tried to boot a scrambled kernel. That is why the Checkpoint reads both large files from inside xv6 with wc.

When mkfs builds the host disk, it now reports something like balloc: first 3532 blocks have been allocated out of 4000: the two guest files take most of the disk.

Step 8: Large files: why this design

The change above is deliberately the smallest one that works. Two other designs are worth knowing, and Question 1.4 asks you about one of them.

Why not add a 14th address? The obvious fix is uint addrs[NDIRECT+2]: keep the singly-indirect block and add a doubly-indirect one after it. But that makes struct dinode 12 + 14 x 4 = 68 bytes. 512 / 68 = 7.5 inodes per block: inodes would straddle block boundaries, which xv6’s IBLOCK() and IPB arithmetic does not handle (mkfs asserts BSIZE % sizeof(struct dinode) == 0 for exactly this reason). You would have to pad the inode to 128 bytes, or waste part of every inode block, and change struct inode in file.h to match. Keeping 13 addresses avoids all of that.

Why not keep three tiers? The classic Unix layout keeps the 13 addresses but uses them differently: 11 direct addresses, one singly-indirect, one doubly-indirect. (The well-known xv6 “large files” exercise does exactly this.) That is strictly better for medium files: a file between about 6 KB and 70 KB (blocks 11 to 138) needs only one pointer block, as in stock xv6, while in our layout any file past 12 blocks needs two, the doubly-indirect block and its first indirect block. It costs one extra disk block per such file and, on a cold cache, one extra block read per bmap().

We chose the simpler two-tier tree anyway. The code has one indirect case instead of two, in three places (bmap(), itrunc(), iappend()), and the address arithmetic is just / and %. The performance cost is mostly hidden by the buffer cache: when a file is read or written sequentially, the same doubly-indirect and indirect blocks are looked up again and again, and they stay in the cache, so the extra read happens once per 128 blocks, not once per block.

Is the doubly-indirect block another layer of inodes? No. It is an ordinary data block, allocated by balloc() like any other, that happens to hold 128 block addresses instead of file data; so is each indirect block below it. The file still has exactly one inode, and addrs[12] in it points to the top of the tree. The inode table, IPB and the on-disk inode format are unchanged (figures/fs-layout.svg).

A side effect on usertests1. Its writetest1 (“big files test”) writes MAXFILE blocks to one file. With the new MAXFILE, that is 16,396 blocks, four times the whole host disk, so the test cannot pass on any disk this small: the kernel panics with balloc: out of blocks. (usertests1 failed on the small disk of unmodified xv6-64 too.) Use usertests2 instead; its bigfile test writes 24 blocks, enough to go through the doubly-indirect tree.

Step 9: The console problem

runxv6 is one xv6 process with one thread. Its main loop must keep calling vmrun(), or the guest stops. In between, it must pass the keys you type to the guest’s serial port (Part 5). With stock xv6, it cannot do both:

• read() of the console blocks. consoleread() sleeps until input is available. If runxv6 called read(0, ...) with nothing typed, the guest would freeze until you pressed a key.
• There is no way to ask first. xv6 has no O_NONBLOCK, no select() and no poll(). It does not have threads either, and a second process cannot help: only the process that created a VM may run it, and the second process’s read() would then have to be delivered over a pipe, whose read() blocks too.
• The console edits lines. Stock consoleintr() echoes each key, handles backspace, Ctrl-U and Ctrl-D, and makes a line visible to read() only when you press Enter. But the guest xv6 runs its own copy of the same console code on its serial port, so it echoes and edits too. Through the stock host console, every key would appear twice, and the guest would get nothing until Enter, so even the guest’s backspace could not work.

The fix has two halves, like a Unix terminal: a raw mode for the console (stty raw on Unix), and a way to ask whether a read would block (a tiny poll()). The next steps show both.

Step 10: Raw mode and consready()

Reference: console.c.

// HW2: raw mode (consraw), for runxv6. Characters are not echoed or
// edited, and read() returns each one as soon as it arrives.
//
// runxv6 passes every key to the guest's serial port, and the guest
// xv6 runs its own console code on it: the guest echoes, handles
// backspace and Ctrl-U, and buffers lines. If the host did that too,
// every key would be echoed twice, and the guest would get keys only
// after Enter. Unix terminals have the same mode (stty raw).
static int rawmode;

// Set by the consraw() system call (sysfile.c). Taking cons.lock
// orders the change with consoleintr() and consoleread(), which test
// rawmode while holding it.
void
consraw(int on)
{
  acquire(&cons.lock);
  rawmode = on;
  release(&cons.lock);
}

// HW2: would a read of the console return without blocking?
// consoleread() sleeps while input.r == input.w, so a read returns at
// once if and only if some committed input is waiting. Characters of a
// line still being edited (between input.w and input.e) do not count.
// Used by the fdready() system call.
int
consready(void)
{
  int r;

  acquire(&cons.lock);
  r = input.r != input.w;
  release(&cons.lock);
  return r;
}

Recall xv6’s console input buffer: a circular buffer with three indices. input.r is where read() takes the next character, input.w is the end of committed input (whole lines), and input.e is the end of what has been typed, including a line still being edited. consoleread() sleeps while input.r == input.w.

consready() simply tests that condition, under the same lock. It is the exact negation of “a read would sleep”, which is what makes it safe to use: if it says yes, a read() returns without sleeping. And the answer stays true until the caller reads, as long as no other process reads the console at the same time, because input only accumulates.

rawmode is a single global flag, set under cons.lock so that it changes atomically with respect to the interrupt handler and readers, which test it while holding the lock.

Step 11: Raw mode in consoleintr() and consoleread()

Reference: console.c.

The keyboard and serial interrupt handlers both call consoleintr() with a function that returns the next input character. In raw mode, every character goes straight into the buffer:

  acquire(&cons.lock);
  while((c = getc()) >= 0){
    if(rawmode){
      // HW2 raw mode: no echo, no editing; every character is
      // available to read() at once.
      //
      // No consputc(): the guest echoes what it wants shown. Control
      // characters are not interpreted: Ctrl-P, Ctrl-U, backspace and
      // Ctrl-D reach the guest like any other byte. '\r' is not turned
      // into '\n', as on a real serial line; the guest's own console
      // code does that conversion.
      //
      // Keep the character only if the buffer has room; otherwise drop
      // it, as line mode does.
      if(input.e-input.r < INPUT_BUF){
        input.buf[input.e++ % INPUT_BUF] = c;
        // Commit it at once (line mode waits for the end of the line);
        // this also commits any half-edited line from before raw mode.
        input.w = input.e;
        // Wake a reader sleeping in consoleread().
        wakeup(&input.r);
      }
      continue;
    }

Three things are skipped: the echo (consputc()), the special keys, and the wait for a newline. Setting input.w = input.e commits each character the moment it arrives, so consready() turns true right away. Even Ctrl-P (xv6’s process listing) is passed on: in raw mode, it lists the guest’s processes, not the host’s.

Reading needs two changes:

    c = input.buf[input.r++ % INPUT_BUF];
    // In raw mode Ctrl-D is data for the guest, not end of file.
    if(c == C('D') && !rawmode){  // EOF
      if(n < target){
        // Save ^D for next time, to make sure
        // caller gets a 0-byte result.
        input.r--;
      }
      break;
    }
    *dst++ = c;
    --n;
    if(c == '\n')
      break;
    // HW2: in raw mode, return as soon as the committed input runs out,
    // even if fewer than n bytes were read. A raw-mode reader wants
    // keys as they are typed, and a single key (with no '\n' to end
    // the line) would otherwise leave read() asleep until n arrived.
    // Since the loop only gets here after taking at least one byte, a
    // raw read still blocks until there is something to return.
    if(rawmode && input.r == input.w)
      break;  // raw mode: return what has arrived, not a whole line
  }

First, Ctrl-D is just a byte, so read() never reports end of file in raw mode. Second, and more important: in line mode, consoleread() loops until it has n bytes or a newline. A raw reader that asks for 64 bytes after one keystroke would sleep, waiting for 63 more, and fdready()’s promise that the read will not block would be broken. So in raw mode the loop stops as soon as the committed input runs out, and read() returns what has arrived: often a single byte.

Step 12: pipeready(), fdready() and consraw()

Reference: pipe.c, sysfile.c.

fdready() should answer for any file descriptor, not only the console. runxv6 reads its keys from file descriptor 0, which is a pipe if you run, say, echo ls | runxv6 xv6kernel xv6fs.img. A pipe gets the same kind of test:

// HW2: would a read of p return without blocking? It does if there is
// data, or if the write end is closed (the read returns 0).
// This is the negation of the condition piperead() sleeps on. Used by
// the fdready() system call.
int
pipeready(struct pipe *p)
{
  int r;

  // Take the lock so that nread, nwrite and writeopen are read as one
  // consistent snapshot, not halfway through a pipewrite().
  acquire(&p->lock);
  // nread and nwrite count bytes ever read and written; they differ
  // exactly when the pipe holds unread data.
  r = p->nread != p->nwrite || !p->writeopen;
  release(&p->lock);
  return r;
}

Note the second half of the test: a pipe whose writers are all gone does not block, it returns 0 (end of file). “Ready” means “a read() returns at once”, not “there is data”.

The system call dispatches on the file’s type:

// int fdready(int fd)
//
// HW2. Returns 1 if a read() of fd would not block, 0 if it would, or
// -1 for a bad fd. runxv6 uses it to poll the console between runs of
// the guest, since xv6 has no non-blocking I/O.
//
// runxv6 is one process with one thread, and it must keep calling
// vmrun() to run the guest. xv6 has no O_NONBLOCK, select() or poll(),
// and read() of the console sleeps until input arrives, so a plain
// read() would stop the guest until the next key. With fdready(),
// runxv6 reads only when a read cannot block. The answer stays true
// until the caller reads: input only accumulates, as long as no other
// process reads the same file.
uint64
sys_fdready(void)
{
  struct file *f;

  // argfd() checks that fd is open in this process.
  if(argfd(0, 0, &f) < 0)
    return -1;
  // A pipe blocks while it is empty and its write end is open.
  if(f->type == FD_PIPE)
    return pipeready(f->pipe);
  // The console blocks until input is committed. Only the console
  // device can block; other devices would need their own test.
  if(f->type == FD_INODE && f->ip->type == T_DEV && f->ip->major == CONSOLE)
    return consready();
  // A regular file or directory never waits for input: readi() returns
  // at once (it may wait for the disk, but that always finishes).
  // f->readable is not checked: on an fd not open for reading, read()
  // fails at once whatever fdready() says.
  return 1;
}

A regular file always counts as ready: reading it may wait for the disk, but that wait always ends, which is the kind of blocking runxv6 does not mind. The other system call just sets the flag:

// int consraw(int on)
//
// HW2. Turn the console's raw mode on or off. In raw mode, typed
// characters are not echoed or edited, and read() returns them as soon
// as they arrive instead of a line at a time. runxv6 turns it on so that
// every key goes straight to the guest, which does its own echoing.
//
// The mode belongs to the console, not to the process: it stays on
// until someone turns it off, so runxv6 turns it off before it exits
// (die() and the Ctrl-] quit), or the shell would get raw input. If
// runxv6 is killed, the console stays raw.
uint64
sys_consraw(void)
{
  int on;

  if(argint(0, &on) < 0)
    return -1;
  // Any nonzero value turns raw mode on.
  consraw(on);
  return 0;
}

This is how runxv6 uses them (you will meet this code again in Part 5): it turns raw mode on before the first vmrun(), and after every return from vmrun() it drains whatever keys are waiting, without ever blocking:

  while(fdready(0) > 0){
    if((n = read(0, buf, sizeof(buf))) <= 0)
      return;

A pitfall: raw mode outlives runxv6. The mode belongs to the console, not to the process, and xv6 has no terminal session to reset it. runxv6 turns it off on every normal way out: when you type Ctrl-] (runxv6: stopped (Ctrl-])) and in die(). But if runxv6 is killed, or crashes in a way that skips die(), the console stays raw. The host shell then gets your keys without echo, backspace no longer edits, and on a serial console (make qemu-nox, make bochs-nox) Enter sends '\r', which raw mode no longer turns into '\n', so the shell never sees the end of a line. Reboot xv6 if that happens. While developing Part 5, quit runxv6 with Ctrl-].

Step 13: The new system calls and vmxapi.h

SDM: 30.2.1 “Basic VM-Exit Information”, Table 30-5 “Exit Qualification for I/O Instructions”; 27.9.1 (the guest-linear address field); Table 27-6 (the “interrupt-window exiting” control); Appendix C, Table C-1 “Basic Exit Reasons”.

HW2 adds four system calls, numbers 26 to 29. The plumbing (syscall.h, syscall.c, usys.S, user.h, defs.h) and the sys_ wrappers are given. Two of them are hypervisor calls, whose kernel halves you write in later parts; two are the console calls of Steps 10 to 12.

runxv6 uses vmmem() to load the guest kernel, to write the BIOS data area and the MP tables (Part 3), and to move disk sectors (Part 5); vminject() for the guest’s timer (vector 32), COM1 (36) and IDE (46) interrupts.

vmxapi.h also gains a few constants:

• EXIT_INTR_WINDOW (7). The exit that interrupt-window exiting causes when the guest becomes able to take an interrupt (Table C-1; the control is bit 2 in Table 27-6). Part 4 turns the control on while an interrupt is pending; the kernel handles the exit itself, so runxv6 never sees it.
• VM_LAPIC_ADDR (0xFEE00000) and VM_IOAPIC_ADDR (0xFEC00000). The guest-physical addresses of the local APIC and the I/O APIC. vmcreate() maps a zeroed page at each (Part 3): xv6 can write their registers and read them back, but nothing behind them is emulated, and no interrupt comes through them.

Finally, struct vmexit gets four fields for string I/O, the INS and OUTS instructions that xv6’s IDE driver uses to move whole sectors (rep insl, rep outsl):

    uchar string;    /* 1 for INS/OUTS (exit qualification bit 4) */
    uchar rep;       /* 1 if it has a REP prefix (qualification bit 5) */
    uint64 count;    /* units to transfer: RCX with REP, else 1 */
    /*
     * The VMCS guest-linear address (GUEST_LINEAR_ADDRESS in vmx.h):
     * RDI for INS, RSI for OUTS, since 64-bit mode ignores the segment
     * bases. It goes through the guest's page tables, not the EPT.
     */
    uint64 addr;     /* guest-virtual address of the memory operand */

For an ordinary IN or OUT, the data is in RAX, and HW1’s io.data and vmsetreg(VMREG_RAX, ...) were enough. For INS and OUTS the data is in guest memory, at a guest-virtual address. The kernel decodes the exit (Part 5): string and rep are bits 4 and 5 of the exit qualification (Table 30-5), count is the guest’s RCX when there is a REP prefix, and addr is the VMCS guest-linear address field, which the CPU fills in for INS and OUTS exits (27.9.1). runxv6 then copies the data with vmmem(..., VMMEM_VIRT), which is why Part 2 has to translate guest-virtual addresses.

struct vmexit describes the CPU, not the devices. Look at what the structure contains: a port number, a size, a direction, a value or a memory address. Nothing in it says “disk” or “serial port”. The kernel reports only what the CPU trapped, in the CPU’s own terms, and it is runxv6 that knows port 0x3F8 is COM1’s data register and 0x1F0 the IDE data port. This is the same split as in KVM, where the kernel fills in an I/O exit in the shared kvm_run structure (direction, size, port, count, data) and QEMU’s device models give the ports a meaning. It keeps the kernel small, and it means that adding a device to the guest changes only the user program. (KVM goes one step further than we do for string I/O: it copies the data between guest memory and the kvm_run page itself, so QEMU never translates guest addresses.)

Checkpoint

Nothing in this part is yours, so the checkpoint checks that the starter tree builds and that the host side works before you start changing the hypervisor.

1. Build. In the starter tree:

make
make fs.img

Watch the output: you should see the guest objects compiled into guest/ (with -DPHYSTOP=0x2000000 -DFSSIZE=2000), ld linking xv6kernel, mkfsguest building xv6fs.img with total 2000, and then mkfs building fs.img:

nmeta 59 (boot, super, log blocks 30 inode blocks 26, bitmap blocks 1) blocks 3941 total 4000
balloc: first 3532 blocks have been allocated

(The exact count of allocated blocks depends on your compiler, but it should be well under 4000.) Check the host-side files: ls -l xv6kernel xv6fs.img fs.img shows about 270 KB, exactly 1,024,000 bytes and exactly 2,048,000 bytes.

2. Boot the host and look at the large files. Boot xv6 (make qemu-nox, or make bochs-nox; see “Where to run it”). At boot it prints sb: size 4000 nblocks 3941 .... At the shell:

$ ls
.              1 1 512
..             1 1 512
README         2 2 1688
...
timertest      2 19 16952
runvm          2 20 21528
runxv6         2 21 35784
helloguest     2 22 16414
xv6kernel      2 23 273608
xv6fs.img      2 24 1024000
console        3 25 0
$ wc xv6kernel
2181 8247 273608 xv6kernel
$ wc xv6fs.img
1921 11968 1024000 xv6fs.img

ls shows the two files with their full sizes (your xv6kernel, built by your compiler, may differ a little in size and in its line and word counts; xv6fs.img is always 1,024,000 bytes). wc reads every byte of each file, through bmap()’s doubly-indirect path for everything past the first 6 KB, so it is a real test of Steps 4 to 7: it must finish, and the last number before the name (the byte count) must equal the size ls shows.

3. Run the file system tests on the host.

$ usertests2
usertests 2 starting
rmdot test
rmdot ok
fourteen test
fourteen ok
bigfile test
bigfile test ok
...
uio test
pid 69 usertests2: trap 13 err 0 on cpu 1 rip 0x170c addr 0x0--kill proc
uio test done
exec test
ALL TESTS PASSED

The trap 13 line is expected: the uio test checks that a user program cannot execute I/O instructions (outb, inb), and the kernel kills the child with a general-protection fault. bigfile writes and reads back 24 blocks, so it crosses from the direct blocks into the doubly-indirect tree, and other tests create and delete many files, which exercises itrunc(). usertests2 refuses to run twice on the same disk (already ran user tests -- rebuild fs.img): run make fs.img again, after deleting fs.img, to start fresh. Do not run usertests1 here: its big-file test cannot pass on this disk (Step 8).

4. HW1 still works.

$ runvm helloguest
Hello from a VT-x guest
runvm: guest halted at rip 0x10004E

runxv6 xv6kernel xv6fs.img will not boot yet: that is Parts 2 to 5.

If something fails:

Questions

Answer these in answers-hw2.md, two or three sentences each (show your arithmetic where there is some).

1.1. What is the largest file, in blocks and in bytes, that xv6 can hold with the layout of Step 4? How many disk blocks does xv6fs.img (1,024,000 bytes) occupy as a file in fs.img, counting its pointer blocks? How large could a file be if BSIZE were 1024 bytes instead of 512?

1.2. Suppose someone changes GUESTMEM in runxv6.c to 16 MB and forgets the Makefile. What happens when the guest boots, at about which guest-physical address, and why? What happens in the opposite case (GUESTMEM 64 MB, PHYSTOP still 32 MB)?

1.3. runxv6 must pass your keystrokes to the guest while it keeps the guest running. Explain why it needs both fdready() and consraw(): what would go wrong with fdready() alone (console still in line mode), and what would go wrong with consraw() alone (no fdready())?

1.4. Why did we turn addrs[12] into a doubly-indirect block instead of adding a 14th address to the inode? Give one advantage the classic 11 direct + singly-indirect + doubly-indirect layout has over ours, and explain why the difference matters little in practice.

1.5. The Makefile builds the guest kernel’s objects in guest/ instead of next to the host kernel’s. Describe what could go wrong, step by step, if both kernels were built in the same directory, and which kernel would end up broken.

Part 2: Reading and writing guest memory

In this part you give the kernel a way to copy bytes between a user buffer in runxv6 and the memory of a VM. It is one new system call, vmmem(), and the kernel functions behind it. When you finish, nothing new appears on the screen: vmmem() is a tool, and the next parts use it. Part 3 loads the guest kernel and writes the firmware tables with it, and Part 5 moves disk data with it. The checkpoint at the end of this part shows you how to test it on its own anyway.

In HW1 the kernel wrote guest memory only once, in vmcreate(), when it copied the image in. Booting xv6 needs much more:

• Loading the kernel and writing firmware tables. runxv6 copies the segments of xv6kernel to the physical addresses in its ELF header, and writes the BIOS data area (address 0x413) and the MP tables (0xF0000). These are addresses in the guest’s physical memory.
• Moving disk data. xv6’s IDE driver reads a sector with rep insl and writes one with rep outsl. The memory operand is b->data, a buffer in the guest kernel. The guest runs with paging on, so this address is virtual: it means something only through the guest’s own page tables.

So vmmem() takes either kind of address, and a flag says which.

Three address spaces

Three kinds of address meet in this part. Keep them apart, because most bugs here are one used in place of another.

1. Guest-virtual addresses are what guest instructions use. The guest translates them with its own 4-level page tables, rooted at its own CR3. We did not build these tables and we do not control them: the guest kernel builds them in its own memory, exactly as on a real machine. For the guest kernel’s data, such as b->data, these addresses are KERNBASE (0x80000000000) plus the guest-physical address, because xv6 maps all of its physical memory there.
2. Guest-physical addresses are what the guest believes is RAM: 0 to 32 MB (GUESTMEM), plus the two zeroed pages at the local APIC and I/O APIC addresses. Our EPT translates them. The guest cannot see or change the EPT.
3. Host-physical addresses are real RAM. The host kernel cannot dereference a physical address; it uses P2V(pa), its own kernel-virtual mapping of all physical memory, as everywhere in xv6.

You saw the first two stages in the HW1 figure, now from the CPU’s point of view:

When the guest runs, the CPU does both stages in hardware on every memory access. When runxv6 asks the kernel to touch guest memory, the guest is not running, so the kernel has to do the same walks in software. That is what this part builds:

• gpa2kva(vm, gpa): guest-physical to kernel-virtual, through the EPT (your HW1 eptwalk(), then P2V());
• gva2gpa(vm, gva, &gpa): guest-virtual to guest-physical, through the guest’s page tables;
• vmmem(id, addr, buf, len, flags): the copy, page by page, using one or both.

Where to look things up. As in HW1, “SDM” is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026). The guest’s page tables are ordinary x86-64 paging, which this edition describes in Chapter 5 of Volume 3A (“Paging”); the EPT is in Chapter 31 of Volume 3C.

What is given and what you write

These pieces are given in the starter code:

• three new fields in struct vm, cr0, cr3 and cr4 (Step 1 explains them), next to the HW1 field efer, which this part also uses;
• the system call: sys_vmmem() in sysvmx.c, its number, the user-space stub (usys.S, user.h) and the prototype of vmmem() in defs.h;
• the flags VMMEM_WRITE and VMMEM_VIRT in vmxapi.h;
• the constants CR0_PG, PTE_P and PTE_PS (mmu.h) and EFER_LMA (vmx.h).

You write:

The three functions are next to each other in vmx.c, in the order gpa2kva(), gva2gpa(), vmmem(), after vmcreate().

Step 1: Save the guest’s control registers at every exit

SDM: 30.3.1 “Saving Control Registers, Debug Registers, and MSRs”; Chapter 33, “VMCLEAR–Clear Virtual-Machine Control Structure” and “VMREAD–Read Field from Virtual-Machine Control Structure”.

To walk the guest’s page tables, the kernel needs three facts about the guest: whether paging is on (CR0.PG), where its top-level table is (CR3), and which paging mode it uses (EFER.LMA). vmrun() already saves EFER after every exit (HW1). Now it saves the control registers too.

In your vmrun(), find the block that reads the guest state after a successful exit. It ends like this:

      /* Saved because of EXIT_SAVE_IA32_EFER; picks up a new LMA. */
      vm->efer = vmread(GUEST_IA32_EFER);
      e.rip = vm->rip;

Add these lines right after e.rip = vm->rip;, still inside the same else branch, before the VMCLEAR:

      /* HW2: state for vmmem(), interrupt injection and string I/O. */
      /*
       * The guest's own control registers, as the exit saved them (SDM
       * 30.3.1): the real values, not the read shadows (GUEST_CR4 has
       * VMXE set).  vmmem() runs after vmrun() has VMCLEARed the VMCS,
       * when VMREAD is impossible, so keep copies.
       */
      vm->cr0 = vmread(GUEST_CR0);
      vm->cr3 = vmread(GUEST_CR3);
      vm->cr4 = vmread(GUEST_CR4);

(Parts 4 and 5 add more lines after these, under the same HW2 comment.)

What the exit saves. While the guest runs, its CR0, CR3 and CR4 are loaded in the real registers. At every VM exit the CPU writes them into the guest-state fields GUEST_CR0, GUEST_CR3 and GUEST_CR4 before it loads the host’s (SDM 30.3.1). So after an exit these fields hold the guest’s registers as they were at the moment it exited. If the guest has switched page tables (xv6 does, on every context switch), GUEST_CR3 has the new one.

Why copies? vmmem() is called by runxv6 through a system call, between two calls to vmrun(). By then vmrun() has executed VMCLEAR, as the HW1 invariant requires (a VMCS must be cleared on the CPU where it was loaded, before the process can move to another CPU). VMCLEAR writes the VMCS out to its page and makes the current-VMCS pointer invalid, and VMREAD with no current VMCS fails (see the operation section of VMREAD in Chapter 33). So vmmem() cannot read the VMCS. It could load it again with VMPTRLD, but then it would have to keep the process on one CPU and clear it again, all to read three values that vmrun() could save for the cost of three VMREADs. So we keep copies in struct vm.

The copies are never stale when vmmem() uses them. A process in xv6 has one thread, and while it is in vmmem() it is not in vmrun(), so the guest is not running and its registers cannot change. runxv6 handles an exit and then calls vmmem(): it sees the registers of exactly the exit it is handling.

The copies are also never written back. The VMCS keeps its own GUEST_CR3 across VMCLEAR and VMPTRLD, and only the guest changes it.

Real values, not read shadows. In HW1 you made CR4.VMXE a host-owned bit, so the guest reads its CR4 through a read shadow without VMXE. GUEST_CR4 is the real register, with VMXE set. For the page walk we want the real values, which is what the guest-state fields hold. gva2gpa() uses cr0 and cr3; cr4 is saved along with them because it is the third control register paging depends on (CR4.PAE and CR4.LA57 select the paging mode), although xv6 never changes those bits.

Step 2: gpa2kva(), guest-physical to kernel-virtual

SDM: 31.3.2 “EPT Translation Mechanism”, Table 31-7 “Format of an EPT Page-Table Entry that Maps a 4-KByte Page”.

Paste this as the body of gpa2kva():

  uint64 *pte;

  /*
   * Walk the EPT in software, as the CPU does for every guest access.
   * alloc = 0: never create tables; eptwalk() returns 0 when a table
   * on the way is missing (gpa is above memsz and is not an APIC page).
   * A leaf entry may exist but be empty, if its page table was created
   * for a neighboring page; no R/W/X bit means "not present".
   */
  if((pte = eptwalk(vm->ept, gpa, 0)) == 0 || !(*pte & EPT_RWX))
    return 0;
  /*
   * EPT_ADDR() keeps the host-physical address of the 4 KB page (bits
   * 51:12), dropping the permission and memory-type bits; P2V() turns it
   * into the kernel's mapping of that page, and the low 12 bits of gpa
   * are the offset in it.  The result is good only up to the end of
   * this page: the next guest page may be anywhere in host memory.
   */
  return (char*)P2V(EPT_ADDR(*pte)) + gpa % PGSIZE;

This reuses the eptwalk() you wrote in HW1, with alloc = 0, so it only looks and never builds. There are two ways for a guest-physical address to be unmapped, and the function checks both:

• A table on the way is missing. eptwalk() returns 0. This happens for most addresses above the guest’s 32 MB that are not one of the two APIC pages.
• The table exists but the leaf entry is empty. Each EPT page table covers 2 MB, so the one that maps an APIC page also has slots for the 511 pages around it, which are not mapped. Those slots are 0. As in HW1, an EPT entry is present if any of its R, W, X bits is set.

If the entry is present, EPT_ADDR() gives the host-physical address of the 4 KB page, P2V() the kernel’s address for it, and the low 12 bits of gpa the byte inside it. The EPT maps 4 KB pages, and a page keeps its offset through translation, so gpa % PGSIZE is also the offset in the host page.

The pointer is good for one page only. In vmcreate() every 4 KB of guest memory got its own kalloc() page. Guest-physical pages 0x100000 and 0x101000 are neighbors for the guest, but their host pages can be anywhere. A caller that wants more than the rest of this page must call gpa2kva() again. vmmem() does.

Why not just P2V(gpa)? Because guest-physical address 0x100000 is not host-physical address 0x100000. P2V(0x100000) is the host kernel’s own code. Every guest-physical address must go through the EPT, the same way the CPU sends every guest access through it.

Step 3: gva2gpa(), the guest page-table walk (Your turn)

SDM: 5.1.1 “Four Paging Modes”; 5.5.2 “Use of CR3 with Ordinary 4-Level Paging and 5-Level Paging” (Table 5-12); 5.5.4 “Linear-Address Translation with 4-Level Paging and 5-Level Paging” (Figures 5-8, 5-9 and 5-10, and Tables 5-15 to 5-20, the entry formats; Figure 5-11 puts all the formats on one page); 31.3.1 “EPT Overview” (how the CPU walks the guest’s tables through the EPT).

You start with:

/*
 * HW2. Translate a guest-virtual address to a guest-physical one, the way
 * the guest's MMU would: walk the guest's own page tables, which live in
 * guest-physical memory, starting at the guest's CR3 (saved at the last
 * exit).  With paging off, the two addresses are the same.  Handles
 * 4-level (IA-32e) paging with 4 KB, 2 MB and 1 GB pages (SDM Vol. 3A,
 * 5.5).  Returns -1 if the address is not mapped.
 */
static int
gva2gpa(struct vm *vm, uint64 gva, uint64 *gpa)
{
  uint64 table, e, size;
  uint64 *p;
  int level;

  return -1;
}

Goal. Translate the guest-virtual address gva to a guest-physical address, the way the guest’s own MMU would at this moment, and store it in *gpa. Return 0 on success and -1 if the address has no translation. The guest is not running, so use the registers saved at the last exit: vm->cr0, vm->cr3 and vm->efer.

(The SDM calls these addresses linear, not virtual. In 64-bit mode the segment bases that turn a “logical” address into a linear one are 0, and the address the CPU gives us for INS and OUTS already includes the base, so for us the two words mean the same thing.)

The contract. Your function must do exactly this:

1. Paging off (CR0.PG = 0): there is no translation. The CPU uses a linear address directly as a physical one (SDM 5.1.1; 31.3.1 says the same for a guest: “each linear address is treated as a guest-physical address”). Set *gpa = gva and return 0. The guest runs like this at the start of entry.S, before it turns paging on.
2. Paging on, but not in long mode (EFER.LMA = 0): that would be 32-bit or PAE paging, whose tables have other formats. xv6 turns on paging and long mode in the same instruction, so it never uses them. Return -1.
3. Paging on and long mode: 4-level paging. Walk it:
   • The PML4’s address is bits 51:12 of CR3 (Table 5-12). The low 12 bits are flags (PWT, PCD), and the top bits can hold other controls. Mask with 0x000FFFFFFFFFF000ULL.
   • There are four levels: PML4, page-directory-pointer table (PDPT), page directory (PD) and page table (PT). If you number them 3, 2, 1, 0, as eptwalk() does, the index into the table at level level is the 9 bits of gva starting at bit 12 + 9*level: bits 47:39, 38:30, 29:21 and 20:12 (Figure 5-8). Each entry is 8 bytes.
   • An entry is present if bit 0 (P, PTE_P) is 1. If it is 0, there is no translation: return -1. (The CPU would raise a page fault in the guest. We just report failure.)
   • In a PDPT entry (level 2) and a PD entry (level 1), bit 7 (PS, PTE_PS) set means the entry maps a large page, 1 GB or 2 MB, and the walk stops there (Tables 5-16 and 5-18). In a PML4 entry and a PT entry, bit 7 means something else, so do not test it there.
   • Otherwise, bits 51:12 of the entry are the guest-physical address of the next table. Mask with 0x000FFFFFFFFFF000ULL again.
   • After the PT entry, the page’s address plus the low 12 bits of gva is the answer.
4. The tables are guest-physical. CR3 and every table address inside an entry are guest-physical addresses. To read an entry, compute the guest-physical address of the entry and turn it into a kernel pointer with gpa2kva(). If that returns 0 (the guest’s CR3 or an entry points outside guest memory), return -1.

You do not check access rights (writable, user, execute-disable; SDM 5.6), and you do not set the accessed and dirty bits (SDM 5.8). This walk serves the monitor, which acts like a device moving data for the guest kernel, not like guest code.

Hints.

• Your HW1 eptwalk() has the same shape: a loop over the levels from 3 down, an index computed from the shift 12 + 9*level, and “follow the entry to the next table”. Here the loop can include level 0, since nothing is created; after the loop, table holds the 4 KB page from the PT entry.
• The address of the entry you want is table (guest-physical) plus 8 times the index. An 8-byte-aligned entry never crosses a page, so one call to gpa2kva() is enough for each entry.
• Read each entry into the local variable e once, then test and mask the copy.
• For a large page, size is the page size: 1 GB at level 2 and 2 MB at level 1, which is 1ULL << (12 + 9*level), the same shift as for the index. The answer is the entry’s address bits at and above bit 30 (or 21), combined with the bits of gva below bit 30 (or 21), the offset inside the large page. Figures 5-9 and 5-10 show this.
• Use 64-bit constants (ULL). 1 << 30 still fits in an int, but masks and shifts near bit 39 and bit 51 do not.

Pitfalls.

• Reading a table with P2V(). P2V(table) is host memory at that address, not the guest’s table. Your walk would then read the host kernel’s memory and return nonsense, or worse, an answer that looks plausible. Always go through gpa2kva(), which goes through the EPT, as the CPU does in hardware (31.3.1: “The guest-physical address of the guest page-directory entry (PDE) is translated through EPT”).
• Not masking the flags. An entry that points to a table has flags in bits 11:0 (P, R/W, U/S, A, …). xv6’s upper-level entries have P, W and U set, so an unmasked entry is the table address plus 7 (plus 0x20 more once the CPU has set the accessed bit), and every index you compute from it is off. Bit 63 is the execute-disable bit and bits 62:52 can hold other values (Tables 5-15 to 5-20); left in, they make the address huge. Mask with 0x000FFFFFFFFFF000ULL, and mask CR3 too.
• The PS bit at the wrong level. In a PT entry bit 7 is PAT, a caching attribute (Table 5-20), and in a PML4 entry it is reserved (Table 5-15). If you test it at every level, a 4 KB page with PAT set is taken for a large page.
• The large-page offset. For a 2 MB page, the offset is 21 bits, not 12. If you take only the low 12 bits of gva, every address in the large page translates into its first 4 KB. Also clear the entry’s low bits below the page size: in a large-page entry bit 12 is PAT and bits 29:13 (or 20:13) are reserved, not address (Tables 5-16 and 5-18).
• Not checking PG and LMA. Early in the guest’s boot, paging is off, and CR3 is 0. A walk from CR3 = 0 reads guest memory at 0 as if it were a PML4.

How to check. Nothing calls gva2gpa() until Part 5. The checkpoint below gives you a test you can run as soon as the guest boots in Part 3.

Step 4: vmmem(), find the VM

SDM: none; this is our own interface (vmxapi.h).

vmmem(id, addr, buf, len, flags) copies len bytes between buf and guest memory at addr. With VMMEM_WRITE the copy goes into the guest (from buf), without it out of the guest (into buf). With VMMEM_VIRT, addr is guest-virtual; without it, guest-physical. It returns len, or -1. Paste the first block, after the declarations:

  /*
   * Only the owner may touch the VM's memory.  The guest is not running
   * now: xv6 processes have one thread, and this one is in a system
   * call, not in vmrun().  buf is the caller's user buffer, which
   * sys_vmmem() checked with argptr(); the kernel can use it directly
   * because it runs on the process's page table.
   */
  if((vm = getvm(id)) == 0 || len < 0)
    return -1;

getvm() is the HW1 helper: it turns the id into a struct vm * and checks that the calling process owns the VM. sys_vmmem() (given) fetches the arguments: addr with argint64(), since a guest address is 64 bits, and buf with argptr(), which checks that all len bytes lie inside the calling process. During a system call the process’s own page table is loaded, so the kernel can use that user pointer directly with memmove().

Nobody else can touch the guest’s memory or its saved registers while we copy: the guest runs only inside vmrun(), and this process is busy in vmmem().

Step 5: vmmem(), the copy, page by page

SDM: none beyond Steps 2 and 3.

Paste the rest of the body:

  /*
   * One page per iteration.  Contiguous guest addresses need not be
   * contiguous in host memory: each 4 KB guest-physical page is its own
   * kalloc() page, and consecutive guest-virtual pages may map to
   * unrelated guest-physical ones.  So translate again at every page
   * boundary.  Even inside a 2 MB or 1 GB guest page, the EPT maps
   * 4 KB pages, so the copy still stops at each 4 KB boundary.
   */
  for(off = 0; off < len; off += n){
    if(flags & VMMEM_VIRT){
      /* Through the guest's page tables, as the guest's MMU would. */
      if(gva2gpa(vm, addr + off, &gpa) < 0)
        return -1;
    } else {
      gpa = addr + off;
    }
    /* Then through the EPT to the host page; 0 if outside the guest. */
    if((k = gpa2kva(vm, gpa)) == 0)
      return -1;
    /* Stay within one guest page: the next one may be elsewhere. */
    /*
     * Translation keeps the low 12 bits, so the page offset of addr +
     * off is also that of gpa and of k.
     */
    n = PGSIZE - (addr + off) % PGSIZE;
    if(n > len - off)
      n = len - off;
    /*
     * A failure on a later page returns -1 with the earlier pages
     * already copied; runxv6 treats any failure as fatal anyway.
     */
    if(flags & VMMEM_WRITE)
      memmove(k, buf + off, n);
    else
      memmove(buf + off, k, n);
  }
  return len;

Why page by page. A buffer that is contiguous for the guest is not contiguous for us, for two reasons, one per translation:

• Guest-virtual to guest-physical. Two neighboring virtual pages can map to any two physical pages; the guest’s page tables decide. (For the guest kernel’s own data they happen to be neighbors, but vmmem() must not count on that.)
• Guest-physical to host-physical. Each 4 KB guest page is a separate kalloc() page (Step 2). This one holds even when the guest uses a 2 MB or 1 GB page: the guest sees one large page, but our EPT still backs it with 4 KB pieces.

So each iteration translates addr + off, copies up to the end of that page, and translates again. Every translation keeps the low 12 bits of the address, so the offset in the page is the same for the guest-virtual, the guest-physical and the kernel address, and n can be computed from addr + off. The first iteration may copy less than a page (from the middle of a page to its end), the last may stop short.

The flags. VMMEM_VIRT adds the first translation; gpa2kva() is always used. VMMEM_WRITE only chooses the direction of memmove(). runxv6 uses three combinations:

Failure halfway. If a later page is unmapped, the earlier pages have already been copied when vmmem() returns -1. A real rep insl that hits an unmapped page would also have stored the bytes before it, and then page-faulted. runxv6 does not try to emulate that page fault: it stops the guest.

Checkpoint

This part has no visible effect on its own. Build and make sure nothing broke:

1. make and make fs.img must build without warnings (-Werror).
2. Boot the host and run runvm helloguest. HW1 must still work: Hello from a VT-x guest and runvm: guest halted at rip 0x10004E. If it does not, look at Step 1: the three new lines must be inside the branch for a successful exit, before the VMCLEAR.

The real tests come later:

• Part 3 loads the guest kernel and writes the BIOS data area and the MP tables with vmmem() and guest-physical addresses. That exercises gpa2kva() and the page-by-page loop. If they are wrong, runxv6 stops with cannot write guest memory, cannot write the BIOS data area or cannot write the MP tables, or the guest crashes as soon as it starts because its code landed in the wrong place.
• Part 5 moves disk sectors with VMMEM_VIRT. That is the first use of gva2gpa(). A walk that fails stops runxv6 with runxv6: INS to an unmapped guest address (or OUTS from an unmapped guest address). A walk that returns a wrong address is worse: the sector goes to the wrong guest page, the guest reads a zeroed buffer or garbage, and panics in its file system code (for example while reading the superblock or the root inode), or later, at random, because you overwrote some other page of its memory.

An optional early test of gva2gpa(). You do not have to wait for Part 5. The guest kernel maps its physical memory at KERNBASE (0x80000000000, the same value as in the host, since both are built from the same memlayout.h), so guest-virtual KERNBASE + 0x100000 must translate to guest-physical 0x100000, where the guest kernel is loaded. Once you have finished Part 3 and the guest starts printing, add a temporary check to your vmrun(), right after the three lines from Step 1:

      /* TEMPORARY test of gva2gpa() (Part 2); remove it afterwards. */
      if(vm->cr0 & CR0_PG){
        if(gva2gpa(vm, KERNBASE + 0x100000, &t) < 0 || t != 0x100000)
          cprintf("gva2gpa: bad translation of KERNBASE+1MB\n");
        else if(!gvaok){
          cprintf("gva2gpa: KERNBASE+1MB -> 0x%x\n", t);
          gvaok = 1;
        }
      }

and declare uint64 t; and static int gvaok; with the other declarations at the top of vmrun(). Run runxv6 xv6kernel xv6fs.img. You should see, mixed in with the guest’s output, exactly one line:

gva2gpa: KERNBASE+1MB -> 0x100000

and never the bad translation line. The test runs at every exit with paging on, so it follows the guest through its boot page table (entrypml4 in main.c, which maps KERNBASE with one 1 GB page; an exit there, such as a host timer tick, tests your large-page case) and then through the page tables kvmalloc() and every process build with 4 KB pages. To see the failure case too, try an address that the guest does not map, such as KERNBASE + 0x10000000000, and check that you get -1. Remove the test when you are done.

If something fails:

Questions

Answer these in answers-hw2.md, two or three sentences each.

2.1. gva2gpa() tests bit 7 (PS) of the entry only at the PDPT and PD levels. What does bit 7 mean in a PML4 entry and in a page-table entry (SDM Tables 5-15 and 5-20)? Describe a concrete guest mapping that a walk testing PS at every level would translate wrongly, and what it would return.

2.2. gva2gpa() reads every page-table entry through gpa2kva(), that is, through the EPT. Why can it not use P2V() on the table address, as xv6’s own walkpglevel() (in vm.c) does for its page tables? Which memory would P2V(table) read? How does the CPU solve the same problem when the guest itself accesses memory (SDM 31.3.1)?

2.3. xv6’s IDE driver reads a sector with insl(0x1f0, b->data, BSIZE/4), 512 bytes into b->data. Suppose b->data starts 256 bytes before the end of a guest page. Describe, iteration by iteration, what vmmem() does with this request. Why would one translation followed by one 512-byte memmove() be wrong, even though the guest kernel maps its data to consecutive guest-physical pages?

2.4. vmrun() saves GUEST_CR3 into vm->cr3 after every exit. Why can vmmem() not simply call vmread(GUEST_CR3) when it needs the value? And why is the saved value never out of date when vmmem() uses it?

Part 3: Booting xv6 in a VM

In this part the guest stops being a hello-world program and becomes xv6 itself. You load the xv6 kernel, an ordinary ELF file, into an empty VM, give it the few things a PC would give it before it starts, and let it boot. When you finish, the guest kernel gets as far as starting its scheduler:

$ runxv6 xv6kernel xv6fs.img
runxv6: booting xv6kernel (type Ctrl-] to stop)
xv6...
cpu0: starting 0
cpu0: no VMX support

and then waits. The first thing its first process does is read the disk, and that needs interrupts (Part 4) and string I/O (Part 5).

xv6 is a much more demanding guest than helloguest. It expects a PC: it looks for the tables a BIOS leaves in memory to learn how many CPUs there are and where their interrupt controllers are, it programs a local APIC and an I/O APIC, it asks CPUID whether the CPU supports VMX (our xv6 enters VMX operation on every CPU, as you made it do in HW1), and it takes its own page faults and system calls through its own IDT. This part handles each of these, in the smallest way that works:

Where to look things up. As before, “SDM” is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026). Section numbers are for that edition. The MP tables come from a different document, the Intel MultiProcessor Specification, version 1.4 (1997), which we call “the MP spec”. xv6’s mp.c cites it too.

What is given and what you write

The guest kernel, xv6kernel, is built from the same sources as the host kernel, with two differences passed on the compiler command line: PHYSTOP is 32 MB (-DPHYSTOP=0x2000000), the memory runxv6 gives the VM, and its file system has 2000 blocks (-DFSSIZE=2000). Its object files go in the directory guest/, so that the two builds do not overwrite each other. make fs.img builds xv6kernel and the guest’s disk image xv6fs.img, and copies both into the host’s fs.img, where runxv6 reads them.

Step 1: Your vmcsinit(): let the guest handle its own exceptions

SDM: 27.6.3 “Exception Bitmap”; 28.2 “Other Causes of VM Exits” (exceptions and triple faults); 29.6 “Event Injection”.

Two changes to the function you wrote in HW1 Part 3. The first one saves the primary processor-based controls in the VM. Find the lines that write the control fields and add the vm->cpuctl line right after the write of CPU_BASED_VM_EXEC_CONTROL:

  vmwrite(PIN_BASED_VM_EXEC_CONTROL, pin);
  vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu);
  /*
   * HW2: keep the base value.  vmxinject() writes this field again
   * before every entry, as cpuctl plus, when needed, interrupt-window
   * exiting; it needs the value without that bit to turn it off.
   */
  vm->cpuctl = cpu;  /* vmrun() adds interrupt-window exiting to it */
  vmwrite(SECONDARY_VM_EXEC_CONTROL, cpu2);

vmxinject(), which you write in Part 4, rewrites this VMCS field before every entry: the base value, plus “interrupt-window exiting” when it needs to wait for the guest to enable interrupts. To turn that bit off again it needs the value without it, and that is the value computed here by adjustctl(). cpuctl is a field of struct vm (uint cpuctl;): if your struct vm does not have it yet, add it. Nothing uses it until Part 4.

The second change replaces HW1’s exception bitmap. Replace the comment and the vmwrite(EXCEPTION_BITMAP, 0xFFFFFFFF) of HW1 Part 3, Step 6 (keep the two page-fault lines) with:

  /*
   * The exception bitmap has one bit per exception vector 0-31; a set
   * bit makes that exception cause a VM exit instead of being
   * delivered through the guest's IDT (SDM 27.6.3).  HW1's hello-world
   * guest had no IDT, so it set every bit: each guest exception then
   * exited with its vector, error code and RIP instead of escalating to
   * a triple fault.  An xv6 guest handles its own exceptions (page
   * faults, the system-call trap), so here no exception exits.  A guest
   * without an IDT that faults now triple faults (EXIT_TRIPLE_FAULT).
   */
  /*
   * With a bit set, the hypervisor would have to hand each such
   * exception back to the guest by injecting it (SDM 29.6), at the cost
   * of a VM exit every time.  With the bitmap clear, the CPU delivers
   * exceptions itself, through the IDT that the guest loaded with LIDT.
   * (INT $64, xv6's system call, is a software interrupt, not an
   * exception, so the bitmap never applies to it; SDM 28.2.)
   */
  vmwrite(EXCEPTION_BITMAP, 0);
  /*
   * For page faults the mask/match pair further filters which error
   * codes exit; with bit 14 clear in the bitmap and mask = match = 0,
   * no page fault exits (SDM 27.6.3).
   */
  vmwrite(PAGE_FAULT_ERROR_CODE_MASK, 0);
  vmwrite(PAGE_FAULT_ERROR_CODE_MATCH, 0);

In HW1 the bitmap was all ones because helloguest had no IDT: any exception it raised would have become a double fault and then a triple fault, with no clue left about the cause. With every bit set, an exception became a VM exit with its vector, error code and RIP.

xv6 is different. It loads its own IDT (idtinit() in mpmain()), and it relies on its exceptions: a user program that dereferences a bad pointer page faults, and xv6’s trap() kills it (a fault in the kernel makes it panic). With a bit set, the exception would exit to the hypervisor instead of reaching the guest’s handler, and the hypervisor would have to inject it back into the guest (SDM 29.6), paying for a VM exit every time. With the bitmap 0, the CPU delivers every exception through the guest’s IDT, as on a real machine, and the hypervisor never hears about it.

What about xv6’s system calls? They use INT $64 (T_SYSCALL in traps.h). That is a software interrupt, not an exception, and the exception bitmap does not apply to it at all (SDM 28.2 lists the instructions it does cover, INT1, INT3, INTO, BOUND and the UD instructions, and adds in a footnote that INT1 and INT3 are the opcodes F1 and CC, “not … INT n”). It never exited, even in HW1.

The price: a guest that faults before it has loaded its IDT now triple faults (EXIT_TRIPLE_FAULT, reason 2), and runxv6 can only tell you the RIP. You will meet this in the Checkpoint.

Step 2: Your vmcreate(): back the APIC pages

SDM: 13.4.1 “The Local APIC Block Diagram” (the registers live at FEE00000H); Table 13-1 “Local APIC Register Address Map”; 13.4.6 “Local APIC ID”; 13.6.1 “Interrupt Command Register (ICR)” (the delivery status bit); 13.8.5 “Signaling Interrupt Servicing Completion” (EOI). For the I/O APIC: Intel’s 82093AA I/O APIC datasheet.

xv6 talks to its interrupt controllers through memory-mapped registers: loads and stores to fixed physical addresses, the local APIC at 0xFEE00000 and the I/O APIC at 0xFEC00000 (VM_LAPIC_ADDR and VM_IOAPIC_ADDR in vmxapi.h). In the VM those are guest-physical addresses, and your vmcreate() maps only guest RAM, 0 to 32 MB. The first access would be an EPT violation.

Add i to the declarations at the top of vmcreate():

  int id, n, off, r, i;

and add this loop right after your loop that maps guest memory (the one that ends with *pte = V2P(mem) | EPT_RWX | EPT_MEMTYPE_WB;), before the code that copies the image in:

  /*
   * HW2: xv6 programs its local APIC and I/O APIC through memory-mapped
   * registers at these guest-physical addresses.  Back each with a zeroed
   * page of ordinary memory: writes are kept and read back, which is all
   * xv6 needs to boot.  This is not an emulation of either device;
   * interrupts are delivered by injection instead (vminject()).
   */
  /*
   * Without these pages, xv6's first access to its local APIC would hit
   * an unmapped guest-physical address and cause an EPT violation exit,
   * which runxv6 treats as fatal.
   */
  for(i = 0; i < 2; i++){
    /* Page 0 is the local APIC, page 1 the I/O APIC. */
    gpa = i == 0 ? VM_LAPIC_ADDR : VM_IOAPIC_ADDR;
    /*
     * Both addresses lie just below 4 GB, far above memsz (at most
     * VM_MAXMEM), so the loop above did not reach them: eptwalk() with
     * alloc = 1 creates the EPT tables on the way to them.
     */
    if((pte = eptwalk(vm->ept, gpa, 1)) == 0 || (mem = kalloc()) == 0)
      goto bad;
    /*
     * Zeroed, every location reads as 0 until xv6 writes it, and then
     * as the value written.  So the local APIC ID reads 0, the APIC id
     * that runxv6's MP table gives the one CPU, and the I/O APIC ID
     * reads 0, the id in the MP table's I/O APIC entry.  The
     * delivery-status bit (12) of the interrupt command register, which
     * xv6 polls after writing that register, reads back as the 0 that
     * xv6 wrote, so the poll ends at once.
     */
    memset(mem, 0, PGSIZE);
    /*
     * A real APIC page is uncached device memory.  This one is ordinary
     * write-back memory: nothing reacts to the guest's stores, so
     * caching them does no harm.
     */
    *pte = V2P(mem) | EPT_RWX | EPT_MEMTYPE_WB;
  }

This maps one zeroed page of ordinary memory at each address. It is not an emulation of either device: nothing reacts to the guest’s stores. It works because of what xv6 actually does with these registers:

• Local APIC ID (offset 0x20). lapicid() reads it, and mycpu() looks the value up in cpus[]. It reads 0, so the MP table of Step 5 must give our one CPU APIC id 0.
• Setup writes. lapicinit() writes the spurious-interrupt, timer, LINT0/LINT1, error and task-priority registers. They are stored and never looked at. The timer it programs never fires: the guest’s timer interrupt comes from runxv6, which injects vector 32 once per host tick (Part 4). The version register reads 0, so lapicinit() skips the performance-counter register.
• Interrupt command register (offset 0x300). lapicinit() writes an “INIT level de-assert” to it and then spins until bit 12, delivery status, reads 0 (“idle”, SDM 13.6.1). The page returns what was written, and xv6 wrote that bit as 0, so the loop ends at once. On a real APIC the bit would clear when the IPI was sent.
• EOI (offset 0xB0). lapiceoi() writes 0 to it at the end of every interrupt handler. A real APIC would then let the next interrupt through. Here the write is simply ignored. That is fine, because interrupts reach the guest by injection (Part 4) and never through this APIC.
• I/O APIC. It has only two registers in memory: an index at offset 0 and a data window at offset 0x10. ioapicinit() writes an index and reads the window: the ID reads 0 (it matches the I/O APIC id in the MP table, so xv6 does not complain), and the version register also reads 0, so xv6 sets up just one redirection entry. All the later writes land in the same two words. Harmless, since no interrupt is routed through it either.

A real hypervisor emulates these devices properly (VT-x even has hardware support for a virtual local APIC, SDM Chapter 32, “APIC Virtualization and Virtual Interrupts”). For booting xv6, plain memory is enough.

vmfree() needs no change: it frees every page the EPT maps, these two included.

Step 3 (Your turn): Emulate CPUID

SDM: 28.1.2 “Instructions That Cause VM Exits Unconditionally”; 28.1 (exits are “fault-like”); 30.3.3 “Saving RIP, RSP, RFLAGS, and SSP”; 30.2.5 “Information for VM Exits Due to Instruction Execution” (the instruction length); Appendix C, Table C-1 (reason 10); 26.6 “Discovering Support for VMX”; Vol. 2A, the CPUID instruction.

Boot the guest after Steps 1 and 2 and it stops right after cpu0: starting 0, with runxv6: unhandled exit 10. Exit reason 10 is CPUID (Table C-1). Look at SDM 28.1.2: CPUID is on the short list of instructions that cause a VM exit every time they run in VMX non-root operation, whatever the VMCS says. The hypervisor has to execute it on the guest’s behalf. The guest runs it in vmxinit(), the function you wrote in HW1 Part 1, which our xv6 calls on every CPU.

First, add the case to your vmxhandle(). Put it after the EXIT_HLT case:

  case EXIT_HLT:
    /* Skip the HLT so a later vmrun() continues after it. */
    vm->rip += inslen;
    return 0;

and before case EXIT_IO::

  case EXIT_CPUID:
    /* HW2: emulate it for the guest; see vmxcpuid(). */
    vmxcpuid(vm);
    /*
     * The exit is fault-like: CPUID did not execute, and the saved RIP
     * points at it (SDM 28.1, 30.3.3).  Skip its 2 bytes (0F A2);
     * otherwise the guest would execute it again and exit forever.
     */
    vm->rip += inslen;
    return 1;

It is handled in the kernel and never reaches runxv6, like RDMSR of EFER. As for every fault-like exit, the guest’s RIP still points at the CPUID (SDM 30.3.3), so the case advances it by the instruction length, and return 1 resumes the guest.

Then write vmxcpuid(). The starter has it, with an empty body, just above vmxhandle().

Goal. Do what CPUID would have done in the guest, with one change: the guest must see a CPU without VMX.

1. The guest’s EAX holds the leaf (which information it wants), and for some leaves (4, 7, 0xB, 0xD, …) ECX holds a subleaf. Both are 32-bit values. Where are the guest’s registers right now? (Not in the CPU: vmx_exit saved them.)
2. Execute the real CPUID instruction with that EAX and ECX. You are in VMX root operation, where CPUID does not exit; it returns the host CPU’s answer. Passing it on means the guest sees the real processor’s features.
3. If the leaf is 1, clear bit 5 of ECX, CPUID1_ECX_VMX in vmx.h. That is the bit HW1’s vmxinit() tests first (SDM 26.6).
4. Store the four results in the guest’s RAX, RBX, RCX and RDX. CPUID writes 32-bit registers, and in 64-bit mode writing a 32-bit register clears bits 63:32 of the 64-bit one. So the upper halves must become 0, not keep their old values.

Why hide VMX? Our hypervisor does not support nested virtualization: it cannot run a hypervisor inside the guest. If the guest saw the VMX bit, its vmxinit() would go on to RDMSR of IA32_FEATURE_CONTROL. There is no MSR bitmap, so every RDMSR exits (SDM 28.1.3; “use MSR bitmaps” in Table 27-6), your vmxmsr() knows only EFER, and runxv6 would stop with unhandled exit 31. With the bit clear, the guest prints cpu0: no VMX support and boots on, which is exactly how HW1 behaves on a CPU without VMX. This is also why the Makefile leaves runvm and runxv6 out of the guest’s disk: they could only fail.

Hints.

• cpuid_insn() near the top of vmx.c is almost what you need, but look at what it passes in ECX. A leaf with subleaves would get the wrong answer. Write your own one-line asm volatile, using cpuid_insn() as the model for the constraints, with the guest’s subleaf as the "c" input.
• Use uint variables for the leaf, the subleaf and the four results. Assigning a 64-bit register to a uint keeps its low 32 bits, and assigning a uint to a uint64 zero-extends it: that is point 4 for free.
• Clear the bit only for leaf 1. The same bit position means something else in other leaves.
• vmxcpuid() does not touch RIP. The EXIT_CPUID case does.
• Do not call vmread() or vmwrite(): the VMCS is no longer current when vmxhandle() runs.

How to check. xv6 executes CPUID exactly once, with leaf 1, in vmxinit(). A temporary cprintf("cpuid %x\n", (uint64)leaf) in vmxcpuid() should print once per guest boot. The guest should then print cpu0: no VMX support. If it prints unhandled exit 31 instead, the VMX bit got through.

Step 4: Load the guest kernel (given)

References: the ELF specification (program headers); xv6’s bootmain.c, entry.S and kernel.ld; the Multiboot Specification (version 0.6.96), its section on the machine state at boot.

runvm loaded a flat binary: vmcreate() copied the image to VM_LOAD_ADDR and that was it. The xv6 kernel is an ELF file, and the kernel does not parse ELF. runxv6 does all the loading in user space. This is the start of its main(), after readfile() has read both files into memory:

  /* An empty VM: the image is loaded from the ELF file instead. */
  /*
   * With len 0, vmcreate() copies nothing to VM_LOAD_ADDR (that is for
   * flat binaries such as helloguest). It allocates GUESTMEM bytes of
   * zeroed guest RAM plus the local APIC and I/O APIC pages, and sets
   * the guest up in flat 32-bit protected mode with paging off, the
   * state a multiboot loader leaves and entry.S expects.
   */
  if((vm = vmcreate(kernel, 0, GUESTMEM)) < 0)
    die("vmcreate failed");
  entry = loadkernel(kernel, ksize);
  /* The segments are in guest memory now; the file is not needed. */
  free(kernel);
  mptables();

readfile() (given) is the obvious function: fstat() for the size, one malloc() buffer, and read() in a loop until it has the whole file. The kernel is about 270 KB, the disk image 1,024,000 bytes (2000 blocks of 512 bytes). Both are bigger than the 70 KB maximum file of stock xv6, which is why HW2 lets files grow larger (the doubly-indirect block in fs.h).

vmcreate(kernel, 0, GUESTMEM) passes a length of 0, so your vmcreate() copies nothing (kernel is just a valid pointer). It creates an empty machine: 32 MB of zeroed guest RAM, the two APIC pages of Step 2, and a VMCS with the guest state of HW1. Then loadkernel() fills the memory:

static uint64
loadkernel(char *image, int size)
{
  struct elfhdr *elf = (struct elfhdr*)image;
  struct proghdr *ph;
  int i;

  /* Check that the header fits in the file before reading it. */
  if(size < sizeof(*elf) || elf->magic != ELF_MAGIC)
    die("the kernel is not an ELF file");
  for(i = 0; i < elf->phnum; i++){
    /* The program headers form an array at file offset phoff. */
    ph = (struct proghdr*)(image + elf->phoff + i * sizeof(*ph));
    /*
     * Skip what is not loadable, and segments with nothing in the file
     * (pure bss): their memory is zero already.
     */
    if(ph->type != ELF_PROG_LOAD || ph->filesz == 0)
      continue;
    /*
     * The file bytes must lie inside the file, and the whole segment,
     * bss included, inside guest memory: the guest would use all memsz
     * bytes of it.
     */
    if(ph->off + ph->filesz > size || ph->paddr + ph->memsz > GUESTMEM)
      die("a kernel segment does not fit");
    /*
     * Without VMMEM_VIRT, the address is guest-physical: copy filesz
     * bytes from the file into guest memory at paddr.
     */
    if(vmmem(vm, ph->paddr, image + ph->off, ph->filesz, VMMEM_WRITE) < 0)
      die("cannot write guest memory");
  }
  /*
   * entry.S sets _start = V2P_WO(entry), the physical address of
   * entry, so the entry point is valid with paging off, as RIP needs.
   */
  return elf->entry;
}

An ELF file starts with a header (struct elfhdr in elf.h). It gives the entry point and the location (phoff) and number (phnum) of the program headers, which describe the segments, the pieces of the file to place in memory. loadkernel() walks them and, for each ELF_PROG_LOAD (PT_LOAD) segment, copies filesz bytes from file offset off to guest-physical address paddr, with vmmem() (Part 2; without VMMEM_VIRT the address is guest-physical). On our build there are two:

  Type  Offset    VirtAddr           PhysAddr   FileSiz  MemSiz
  LOAD  0x001000  0x0000080000100000 0x00100000 0x0c370  0x0c370  R E
  LOAD  0x00e000  0x000008000010d000 0x0010d000 0x04eba  0x14100  RW

(readelf -l xv6kernel shows yours.) The first is the code and read-only data, the second the data and the bss.

Where is the guest kernel loaded?

All of the loading happens in user space, in runxv6.c. The kernel’s part is generic: vmcreate() makes empty memory, and vmmem() copies bytes into it. A few details are worth understanding:

• paddr, not vaddr. kernel.ld links the kernel at the high virtual address KERNLINK (0x80000100000, KERNBASE plus 1 MB), but its AT(0x100000) asks for it to be loaded at physical address 1 MB. The two addresses end up in the program header as vaddr and paddr. The guest starts with paging off, where every address is physical, so the code has to be where its physical address says. entry.S then builds page tables that map the high virtual addresses onto the low physical ones and jumps up. Copying to vaddr would not even work: 0x80000100000 is far outside the guest’s 32 MB.
• The bss is already zero. The second segment occupies memsz (0x14100) bytes but only filesz (0x4eba) come from the file. The rest is the bss (uninitialized globals, and the boot stack stack of entry.S), which C requires to start as zero. loadkernel() does not clear it because vmcreate() handed out zeroed pages. It does check that the whole memsz fits in guest memory.
• The entry point is physical. elf->entry is _start, which entry.S defines as V2P_WO(entry): the physical address of entry, 0x10000C on our build (the first 12 bytes are the multiboot header). That is the RIP the guest starts at, with paging off.
• No BIOS, no boot sector. On a PC, the BIOS loads the boot sector (bootasm.S), which switches to 32-bit protected mode and calls bootmain(), which reads the kernel from the disk and jumps to its entry. runxv6 skips both. xv6-64 is multiboot-compatible (it has a multiboot header, so GRUB can boot it), and _start expects what a multiboot loader provides: 32-bit protected mode, flat segments, paging off. That is exactly the state your HW1 vmcsinit() gives every guest (HW1 Part 3, Steps 9 to 12). So loadkernel() mirrors bootmain() almost line for line: the same loop over the program headers, the same copy to paddr, and the same jump to elf->entry, which here is a vmsetreg() of RIP (Step 6).

Notice what we do not give the guest: a memory map. A real BIOS reports which physical memory is RAM (the E820 table) and a boot loader passes it on. xv6 never asks. It assumes RAM from 0 up to PHYSTOP and gives all of it to kalloc(). That is why the guest kernel is built with PHYSTOP equal to GUESTMEM: with a larger PHYSTOP, the guest would sooner or later use memory the EPT does not map.

Step 5 (Your turn): The MP tables

References: the Intel MultiProcessor Specification 1.4, Chapter 4, “MP Configuration Table” (4.1 the floating pointer, 4.2 the configuration table header, 4.3.1 processor entries, 4.3.3 I/O APIC entries); xv6’s mp.c, mp.h and acpi.c.

With Steps 1 to 4 done and mptables() still empty, the guest dies at once: runxv6: guest triple fault at rip 0x8000010...., before it prints anything. (Read on for why.) Even once it gets further, xv6’s main() calls mpinit() third, and mpinit() panics with “Expect to run on an SMP” unless it finds a description of the machine’s CPUs and interrupt controllers in memory.

Where xv6 looks. mpinit() first calls acpiinit(), which scans for an ACPI “RSD PTR " signature (in the EBDA, then 0xE0000 to 0xFFFFF). Our guest memory has none, so it returns 0 and mpinit() falls back to the older MP tables. mpsearch() looks for an MP floating pointer structure, signature _MP_, on a 16-byte boundary, in the three places the MP spec allows, in order:

1. the first KB of the extended BIOS data area (EBDA), whose segment is the 16-bit word at physical 0x40E, in the BIOS data area;
2. if that word is 0, the last KB of base memory, whose size in KB is the 16-bit word at 0x413;
3. the BIOS ROM area, 0xF0000 to 0xFFFFF. (The comment in mp.c says 0xE0000, but the code searches from 0xF0000.)

Our guest’s memory is all zeros, so the word at 0x40E is 0 and xv6 goes to case 2. If the word at 0x413 is also 0, base memory is “0 KB” and xv6 computes its last KB as starting at 0 - 1024: as a 64-bit address that is just below 2^64, and P2V() turns it into an unmapped address just below KERNBASE. The first read page faults, and since xv6 has not loaded its IDT yet (that happens in mpmain(), much later), the fault becomes a triple fault. That is what the empty mptables() produces, with RIP in memcmp(). So the first thing to write is the base memory size: 640 (KB), at 0x413, as a PC BIOS does. Then xv6 searches 0x9FC00 to 0x9FFFF, finds zeros, and moves on to the ROM area.

What to build. Three structures, as the figure shows. Their layouts are in mp.h (runxv6.c includes it), and they are the ones the guest’s mp.c parses, so read mpsearch1(), mpconfig() and mpinit() to see which fields matter.

• The floating pointer (struct mp, 16 bytes, MP spec 4.1) at 0xF0000. Signature _MP_; physaddr, the physical address of the configuration table; length 1 (in 16-byte units); specrev 4 (spec 1.4); type 0, meaning “a configuration table is present” (a nonzero value would select one of the spec’s default configurations, which xv6 rejects); and imcrp 0, meaning no IMCR, so mpinit() leaves ports 0x22 and 0x23 alone.
• The configuration table header (struct mpconf, 44 bytes, MP spec 4.2) at 0xF0010, right after the floating pointer. Signature PCMP; length, the size in bytes of the header and all entries; version 4 (mpconfig() accepts 1 or 4); entry, the number of entries; lapicaddr, the local APIC address, VM_LAPIC_ADDR (xv6 sets its lapic pointer from it). product is free text.
• Directly after the header, the entries: one processor entry (struct mpproc, 20 bytes, type MPPROC, MP spec 4.3.1), enabled (flags bit 0) and the bootstrap processor (MPBOOT); and one I/O APIC entry (struct mpioapic, 8 bytes, type MPIOAPIC, MP spec 4.3.3), enabled, at VM_IOAPIC_ADDR.
• In the BIOS data area, the base memory size (above).

A real table also has bus entries and interrupt assignment entries, which say which I/O APIC pin each device uses. mpinit() skips them (it assumes IRQ n is pin n), and we inject interrupts by vector anyway, so you need none.

The ids must be 0. The processor entry’s APIC id must be the value the guest reads from its local APIC’s ID register, because mycpu() finds the current CPU by looking lapicid() up in cpus[], and panics with “unknown apicid” if nothing matches. The local APIC is the zeroed page of Step 2, so the ID reads 0. Likewise the I/O APIC entry’s id should be 0, the value ioapicinit() reads from the zeroed I/O APIC page.

Checksums. Both structures carry a checksum byte, chosen so that all their bytes add up to 0 modulo 256: the floating pointer’s 16 bytes, and the configuration table’s length bytes (header plus entries). mpsearch1() and mpconfig() both check. The helper checksum(p, len) (given) returns the right byte provided the checksum field is still 0 when you call it. So fill in everything else first and compute the checksum last.

Writing it. Build the structures in local variables in runxv6, then copy them into the guest with vmmem(vm, addr, buf, len, VMMEM_WRITE) (guest-physical, since there is no VMMEM_VIRT). vm is the global that main() set. die() if a vmmem() fails. 0xF0000 is plain RAM in our VM, with no ROM there, and xv6 never hands memory below 1 MB to kalloc(), so the tables stay intact while the guest runs.

Hints.

• The configuration table is the header immediately followed by its entries, and length and the checksum cover all of it. Put the three in one local struct { struct mpconf conf; struct mpproc proc; struct mpioapic ioapic; }: the mp.h structures have no padding (44 + 20 + 8 = 72 bytes), so this struct is byte for byte the table the guest parses, and you can checksum and copy it in one piece.
• memset() each structure to 0 first. Every field you do not set (the OEM table pointer, the extended table length, type, imcrp, the CPU signature) is then 0, which means “absent” or “default”.
• signature and product are byte arrays, not strings: copy into them with memmove(), without a terminating 0.
• physaddr must not be 0: mpconfig() takes 0 to mean a default configuration and gives up.
• The floating pointer must start on a 16-byte boundary: mpsearch1() looks only at every 16th address.
• version values for the local APIC (0x14) and the I/O APIC (0x11) are what real hardware reports. xv6 ignores them.

How to check. Before you boot, convince yourself the tables are right: in a temporary debug block in runxv6, read them back with vmmem() (without VMMEM_WRITE), and check that the 16 bytes at 0xF0000 and the length bytes at 0xF0010 each add up to 0. Then boot (see the Checkpoint). cpu0: starting 0 means mpinit() and mycpu() were happy. If there is no output at all after runxv6: booting, see the Checkpoint for why that is what an mpinit() panic looks like.

Step 6: Start the guest (given)

SDM: as in HW1 Parts 5 and 6.

The rest of main() is given. After mptables():

  /*
   * Start at the kernel's entry point. The other registers start at 0;
   * a multiboot loader would put a magic number in EAX and an info
   * pointer in EBX, but xv6's entry.S does not look at them.
   */
  vmsetreg(vm, VMREG_RIP, entry);

  printf(1, "runxv6: booting %s (type Ctrl-] to stop)\n", argv[1]);
  /*
   * Raw mode: every key goes straight to us, and on to the guest,
   * whose own console echoes and edits lines. In normal mode, our
   * console would echo each key a second time and give us only whole
   * lines.
   */
  consraw(1);

• vmsetreg(vm, VMREG_RIP, entry) sets the guest’s RIP to the physical entry point of Step 4. Every other register starts at 0. A multiboot loader would also pass a magic number in EAX and a pointer to a boot information structure in EBX, but xv6’s entry.S ignores both.
• consraw(1) (a new system call, given) puts the host’s console in raw mode. Normally the host’s console echoes each key and hands read() whole lines. The guest runs its own console code, which echoes and edits lines itself; with the host doing it too, every key would show up twice, and the guest would see nothing until Enter. die() turns raw mode off again, and so does Ctrl-].
• Then the run loop, much like runvm’s: vmrun(), and a switch on the exit reason. EXIT_IO goes to doio(), which picks the device by port: COM1 (0x3F8 to 0x3FF, the guest’s console), the IDE disk (0x1F0 to 0x1F7 and 0x3F6) and the CGA’s CRT controller (0x3D4 and 0x3D5). A read of any other port returns all ones, as on a PC with nothing there. EXIT_TRIPLE_FAULT, EXIT_EPT_VIOLATION and every unknown reason print a message and stop. After each exit, the loop injects the guest’s timer interrupt once per host tick and passes typed keys to COM1; those are Part 4 and Part 5.

The devices, including the CRT controller, which matters even though nobody looks at the guest’s screen, are explained in Part 5. For this part, two facts are enough: the guest’s cprintf() output reaches us through COM1, and only after uartinit() has found the UART (its line xv6... is the sign).

Checkpoint

Build the host’s disk, which now includes xv6kernel and xv6fs.img, and boot the host:

make fs.img
make qemu-nox        # or: make bochs-nox

At the host’s shell prompt, type:

$ runxv6 xv6kernel xv6fs.img
runxv6: booting xv6kernel (type Ctrl-] to stop)
xv6...
cpu0: starting 0
cpu0: no VMX support

Under Bochs this can take a minute. Note what is printed by whom: runxv6: booting comes from runxv6; the other three lines are the guest kernel’s, sent through its emulated serial port. Then nothing more happens. The guest is not hung: its scheduler is running, and its first process, init, is waiting for the disk interrupt of its first read. Type Ctrl-] to stop it (runxv6: stopped (Ctrl-])). If runxv6 does not react, quit the emulator (Ctrl-A x) and boot again.

helloguest must still work: runvm helloguest prints its greeting. It never faults, so the exception bitmap does not matter to it, and it does not touch the APIC pages.

If something fails:

Questions

Answer these in answers-hw2.md, two or three sentences each.

3.1. Before Step 1, the exception bitmap was all ones. Suppose you had kept it that way for the xv6 guest. What would happen the first time a user program in the guest dereferences a null pointer, and what would the hypervisor have to do to make the guest behave exactly as on a real machine? Why does INT $64 not raise the same issue?

3.2. CPUID causes a VM exit unconditionally, while instructions such as HLT exit only if the VMCS asks for it (SDM 28.1.2 and 28.1.3). Why would a CPU designer make CPUID always exit? Use what you did in Step 3 as an example, and give one more reason a hypervisor might want to change what the guest sees.

3.3. loadkernel() copies each segment to p_paddr. What would happen, exactly, if it used p_vaddr instead? And what would go wrong if vmcreate() did not zero guest memory, given that loadkernel() copies only filesz bytes of each segment?

3.4. Your MP table describes one processor with APIC id 0. (a) If its APIC id were 1, where exactly would the guest stop, and why would you see no message? (b) Suppose instead you listed two processors, with ids 0 and 1. Read startothers() in main.c and lapicstartap() in lapic.c: what would the guest do, and why can our zeroed APIC page never make it work?

Part 4: Interrupts

At the end of Part 3 the guest boots far enough to print its first lines, reaches its scheduler, starts its first process, and then goes silent. That process asked the disk for the superblock and went to sleep until the disk interrupt arrives. Nobody delivers it: so far your hypervisor has no way to interrupt the guest. The timer is missing too, so the guest has no clock, no sleep() and no preemption.

In this part you add interrupts for the guest. runxv6 plays the hardware and decides when an interrupt happens; it calls vminject(vm, vector), which only marks the vector pending. Your kernel then delivers it through the guest’s own IDT, at a VM entry, as soon as the guest can take it. When you finish, the guest gets a timer interrupt on every host tick and its first disk interrupt, and it stops one step later, at the first disk read, which needs the string I/O of Part 5.

Where to look things up. As before, “SDM” is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026). This edition renamed the two fields this part is about. The field that VM entry reads to inject an event, called “VM-entry interruption information” in older editions and in most hypervisor code (and in our vmx.h), is now the injected-event identification. The field that a VM exit fills when it interrupts the delivery of an event, called “IDT-vectoring information”, is now the original-event identification. The handout gives both names.

Two interrupt stories

Two quite different things are called “interrupts” in a hypervisor. Keep them apart.

Story 1: host interrupts while the guest runs (HW1, plus one change). The host’s own devices, most importantly each CPU’s local APIC timer, keep interrupting while the guest runs. In HW1 you set “external-interrupt exiting”, so every such interrupt causes a VM exit (reason 1, EXIT_EXTERNAL_INTR; SDM 28.2). The exit does not acknowledge the interrupt: it stays pending in the host’s local APIC (SDM 30.1), and the host takes it through its normal IDT and trap() when vmrun() calls popcli(). The guest never sees host interrupts.

HW2 adds one thing to this story. runxv6 is an ordinary single-threaded process, and while vmrun() loops in the kernel, runxv6 does not run. In HW1 that was fine, because the hello-world guest did I/O all the time. An xv6 guest is different: when it has nothing to do, it spins in its scheduler loop, with no I/O and no HLT, so it never exits to user space. But runxv6 is the guest’s timer and keyboard: it must run regularly to inject the timer interrupt and to look for typed keys. So from now on, vmrun() returns to user space at the first external-interrupt exit after the host’s tick count (ticks) has changed. vmrun() records ticks in vm->starttick when it is called, and the EXIT_EXTERNAL_INTR case of vmxhandle() compares. runxv6 therefore runs at least once per host tick, however quiet the guest is.

Story 2: interrupts for the guest (new). Every interrupt the guest receives is invented by runxv6, which emulates the guest’s hardware, and delivered by your kernel with VMX event injection (SDM 29.6). There are exactly three:

The keyboard part and the disk part are Part 5. This part builds the path that all three take from vminject() to the guest’s trap().

How a PC delivers an interrupt, and what our guest has instead

On a real PC (SDM 13.1 gives the overview), a device raises an interrupt like this:

 device --IRQ pin--> I/O APIC --message--> local APIC --> CPU core
                     redirection table:    IRR, ISR, TPR:    takes it when
                     vector, destination,  picks the         RFLAGS.IF = 1,
                     mask, trigger mode    highest-priority  through IDT[vector]
                                           pending vector

The I/O APIC turns IRQ pin n into a message that carries a vector and a destination CPU; xv6’s ioapicenable(irq, cpu) programs vector 32 + irq. The local APIC of that CPU records the vector in its interrupt request register (IRR, one bit per vector, SDM 13.8.4), picks the highest-priority one (SDM 13.8.3), and hands it to the core when the core has interrupts enabled. The handler’s write to the EOI register tells the local APIC that the interrupt is done. The local APIC also has its own timer, which xv6 programs to fire vector 32 periodically. The old 8259A PIC is still there on a PC, but xv6’s picinit() masks it with two outbs and never uses it.

Our guest has none of these devices. It has:

• No PIC. xv6’s two writes to ports 0x21 and 0xA1 exit to runxv6, which has no device there and drops them, as a PC bus drops a write that no device claims.
• A local APIC that is a zeroed page. vmcreate() (Part 3) maps one page of ordinary memory at 0xFEE00000. Everything xv6 writes there stays, and everything it never wrote reads as 0. That is enough for xv6:
  • the ID register reads 0, which is the APIC id that runxv6’s MP table gives the one CPU, so mycpu() finds its cpus[] entry;
  • lapicinit() sends a broadcast “INIT level de-assert” message and then polls the delivery-status bit (bit 12) of the interrupt command register (ICR) until it is 0; the ICR reads back what xv6 wrote, which has bit 12 clear, so the poll ends at once;
  • the version register reads 0, so lapicinit() skips the performance-counter entry;
  • the writes to the timer registers (TIMER, TDCR, TICR), the EOI register, the TPR and the rest are kept in memory and have no effect. In particular, the local APIC timer never fires.
• An I/O APIC that is a zeroed page at 0xFEC00000. The I/O APIC is programmed through a register-select word and a data window. With plain memory behind them, the data window reads back whatever was last written to it, whatever register was selected. The first read (the version register) returns 0, so ioapicinit() thinks there is one redirection entry; the ID read returns 0, which matches the I/O APIC id in the MP table, so it does not complain. The ioapicenable() calls of the UART and IDE drivers write the data window, and nothing reads it again.
• Injection instead of routing and delivery. vminject() and vmxinject() replace the whole I/O APIC to local APIC to CPU path: runxv6 names the vector directly, and the CPU delivers it through the guest’s IDT as part of VM entry. We use the vectors that xv6 programmed into its I/O APIC and local APIC (32 + IRQ), so the guest’s trap() cannot tell the difference. The pending set in struct vm plays the role of the IRR.

What this gives up:

• EOI, masking and priority control. The guest’s EOI writes, its redirection-table masks and its TPR are ignored. We deliver a vector as soon as the guest has RFLAGS.IF = 1, even if the guest is still in the handler for a lower vector and has re-enabled interrupts. xv6 does neither, so it does not notice.
• Inter-processor interrupts, and so a second CPU. Writes to the ICR go to plain memory; no IPI is ever sent. The guest has one CPU (the MP table lists one).
• The guest’s own timer rate. xv6’s write to TICR has no effect: the guest’s clock ticks at the host’s rate, once per host tick, and only when runxv6 gets to run. Ticks that come while runxv6 waits for the CPU merge into one (see Step 2), so under load the guest’s clock runs slow.

A later homework replaces the zeroed pages with real emulation: leave the APIC pages out of the EPT and emulate each access in the EPT-violation exit, or use the VMX features made for this, “virtualize APIC accesses” (APIC-access exits, SDM 32.4) and virtual-interrupt delivery (SDM 32.2), described in SDM Chapter 32, “APIC Virtualization and Virtual Interrupts”.

What is given and what you write

The stub of vmxinject() in the starter tree injects nothing and never asks for an interrupt-window exit:

static void
vmxinject(struct vm *vm)
{
  vmwrite(VM_ENTRY_INTR_INFO_FIELD, 0);
  vmwrite(CPU_BASED_VM_EXEC_CONTROL, vm->cpuctl);
}

Steps 3 and 4 are the “paste these lines into your own function” kind. Step 5 is a Your turn: you get the rules, the fields and the hints, but no code.

Step 1: The injection state in struct vm

SDM: 27.4.2 “Guest Non-Register State” (Table 27-3); 27.9.3 “Information for VM Exits That Occur During Event Delivery” (Table 27-21); 13.8.4 “Interrupt Acceptance for Fixed Interrupts” (the IRR).

The starter tree adds these fields to struct vm in vmx.c, each with a long comment that you should read there. They hold what vmxinject() needs on the next entry, saved at the last exit, because after vmrun() VMCLEARs the VMCS no vmread() is possible.

cpuctl is set by one line in vmcsinit(), right after the write of the primary controls. You added it with the other vmcsinit() changes in Part 3; check that it is there:

  vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu);
  /*
   * HW2: keep the base value.  vmxinject() writes this field again
   * before every entry, as cpuctl plus, when needed, interrupt-window
   * exiting; it needs the value without that bit to turn it off.
   */
  vm->cpuctl = cpu;  /* vmrun() adds interrupt-window exiting to it */

Without it vm->cpuctl is 0, and the stub’s first VM entry fails with VM-instruction error 7: the required 1-bits of the primary controls are missing.

Step 2: vminject() and nextpending()

SDM: 13.8.3 “Interrupt, Task, and Processor Priority”; 13.8.4 “Interrupt Acceptance for Fixed Interrupts”.

Both are given. vminject() is the system call behind runxv6’s vminject(vm, vector). After checking the VM and the vector, it does one thing:

  vm->pending[vector / 64] |= 1ULL << (vector % 64);

It does not touch the VMCS. It cannot: the caller is runxv6, in a system call, so the guest is not running and no VMCS is current. The interrupt is only recorded, and the next VM entry picks it up. Vectors 0 to 31 are rejected: they are reserved for exceptions (SDM 13.8.3), and the guest’s IDT handles them as such.

nextpending() chooses which pending vector goes first:

/*
 * The highest pending vector, or -1 if none.  A local APIC also delivers
 * the highest-priority pending interrupt first, and an interrupt's
 * priority is its vector: the higher the vector, the higher the priority
 * (SDM 13.8.3, "Interrupt, Task, and Processor Priority").  We have no
 * task priority or in-service register, so this is all of our priority.
 */
static int
nextpending(struct vm *vm)
{
  int i, b;

  /*
   * Scan from the top: word 3 holds vectors 192-255, word 0 vectors
   * 0-63.  Skip empty words, then look for the highest set bit.
   */
  for(i = 3; i >= 0; i--)
    if(vm->pending[i])
      for(b = 63; b >= 0; b--)
        if(vm->pending[i] & (1ULL << b))
          return 64*i + b;  /* bit b of word i is vector 64*i + b */
  return -1;
}

Why the highest vector first? That is what a local APIC does. SDM 13.8.3: the priority class of an interrupt is bits 7:4 of its vector, and within a class, the higher bits 3:0 win. Together that is simply “the higher vector has the higher priority”. So with the disk (46), a key (36) and the timer (32) all pending, the disk goes first. A real local APIC would also compare with its task priority register (TPR) and with the vector in service (ISR) and hold lower-priority interrupts back until the handler’s EOI. We have neither, so the vector order is all the priority we model.

A vector pending twice is delivered once. The set has one bit per vector, so calling vminject(vm, 32) twice before the guest takes the first one leaves one pending timer interrupt. A local APIC behaves the same way: requests for a vector that is already in the IRR collapse into its one bit (SDM 13.8.4). For a device that is harmless, because the handler reads the device’s state and finds everything that happened (uartintr() reads until the receive queue is empty). For the timer it means a lost tick.

Step 3: Additions to your vmrun()

SDM: 30.3.4 “Saving Non-Register State” (the interruptibility state is saved on every exit); 27.9.3 and 30.2.4 “Information for VM Exits During Event Delivery” (the original-event fields); 30.2.5 “Information for VM Exits Due to Instruction Execution” (the instruction length for such exits).

Three additions, in the vmrun() you fixed in HW1.

(a) Before the loop, record the host’s tick count:

  if((vm = getvm(id)) == 0)
    return -1;
  /*
   * HW2: once per call, not per entry: vmxhandle() returns to user
   * space at the first external-interrupt exit after ticks moves on.
   */
  vm->starttick = ticks;

  for(;;){

ticks is the host’s tick counter from trap.c, declared in defs.h. It is set once per call, not per entry: the loop may enter and leave the guest many times, and we want to return when a tick has passed since runxv6 last ran.

(b) Just before the entry, after the VM-entry controls are written:

    if(vm->efer & EFER_LMA)
      entryctl |= ENTRY_IA32E_MODE_GUEST;
    vmwrite(VM_ENTRY_CONTROLS, entryctl);
    /*
     * HW2: set up event injection last, right before the entry, with
     * the VMCS current and interrupts off.  It depends on the RFLAGS
     * written above, and on anything vminject() queued since the last
     * entry.
     */
    vmxinject(vm);

The comment and the call r = vmx_enter(&vm->regs, 0); follow, as before. The position matters: vmxinject() uses vmwrite(), so the VMCS must be current (after vmptrld()), and it decides based on vm->rflags, which must be the value this entry loads into GUEST_RFLAGS.

(c) After each exit, in the branch that handles a real exit (not a failed VMLAUNCH), next to the control registers you save since Part 2:

      vm->cr4 = vmread(GUEST_CR4);
      /*
       * STI and MOV SS blocking at the time of the exit (SDM 30.3.4),
       * for vmxinject() on the next entry.
       */
      vm->intrstate = vmread(GUEST_INTERRUPTIBILITY_INFO);
      /*
       * The original-event identification.  Read it after every exit,
       * before VMCLEAR: the next exit overwrites it, and a valid event
       * not re-injected is lost (see vmxinject()).
       */
      vm->idtvec = vmread(IDT_VECTORING_INFO);
      /*
       * The error code and instruction length mean something only for
       * a valid event (SDM 30.2.4): the error code if bit 11 is set,
       * the length for software events.  For an exit during delivery
       * of a software interrupt or exception, the VM-exit instruction
       * length is that of the INT instruction (SDM 30.2.5).
       */
      if(vm->idtvec & INTR_INFO_VALID){
        vm->idterr = vmread(IDT_VECTORING_ERROR_CODE);
        vm->idtlen = inslen;
      }
    }

The closing brace is the end of that branch, just before vmclear(). (Part 5 adds two more lines here.)

What each read is for:

• intrstate: right after STI (with IF previously 0) or MOV SS, the CPU holds interrupts off for one more instruction, even though RFLAGS.IF may already be 1 (Table 27-3). The exit saves that state (SDM 30.3.4), and vmxinject() must respect it.
• idtvec: if this exit happened in the middle of delivering an event to the guest, this is the only record of that event. Read it after every exit, even ones you think cannot have it: the next exit overwrites it.
• idterr and idtlen: meaningful only when idtvec is valid. inslen is the VM-exit instruction length that vmrun() already reads; for an exit during the delivery of a software interrupt (INT n) it is the length of that INT (SDM 30.2.5).

Step 4: Additions to your vmxhandle()

SDM: 28.2 “Other Causes of VM Exits” (external interrupts, interrupt window); 30.1 “Architectural State Before a VM Exit” (the interrupt is not acknowledged); 30.3.3 “Saving RIP, RSP, RFLAGS, and SSP”; Appendix C, Table C-1 (reason 7, “Interrupt window”).

(a) The external-interrupt case returns to user space once per tick. Your HW1 case only returned 1. Add the if and the two comments above it, so that the case reads:

  case EXIT_EXTERNAL_INTR:
    /*
     * HW2: once the host timer has ticked since vmrun() was called,
     * return to user space, so that runxv6 gets to run at least once per
     * tick: it polls for console input and injects the guest's timer
     * interrupt.  Otherwise just resume the guest.
     */
    /*
     * runxv6 is an ordinary single-threaded process: while vmrun()
     * loops in the kernel, it cannot run.  If vmrun() returned only on
     * I/O or HLT, a guest computing in a loop would get no timer
     * interrupts (no preemption, no sleep() wakeups) and no typed keys.
     * Returning on every host interrupt would also work, but would
     * cost a trip to user space each time; once per tick is enough.
     * (Each CPU's local APIC timer interrupt causes one of these exits,
     * so one comes about once per tick even in an idle host.)
     * Host scheduling does not depend on it: the timer interrupt taken
     * at popcli() in vmrun() can already make this process yield().
     *
     * ticks is advanced by CPU 0's timer interrupt (trap.c), possibly
     * just now at popcli().  It is read without tickslock: a stale value
     * only delays the return to a later exit.
     */
    if(ticks != vm->starttick)
      return 0;
    /*
     * The interrupt is still pending in the local APIC (we don't
     * acknowledge it on exit), so the host took it at popcli().
     * Nothing to do but resume the guest.
     */
    return 1;

Note the order of events: the exit happens, vmrun() VMCLEARs and calls popcli(), the host takes the timer interrupt (which may increment ticks), and only then is vmxhandle() called. So if the interrupt behind this exit was the tick that advanced ticks (only CPU 0’s timer does that), the test already sees it. On another CPU, CPU 0 advances ticks meanwhile, and the next external-interrupt exit sees the change. Returning 0 hands the exit to runxv6, whose main loop treats EXIT_EXTERNAL_INTR as “nothing to emulate, but you get to run now” (Step 6).

(b) A new case for the interrupt-window exit. Add it inside the switch, before default (the reference puts it after EXIT_HLT):

  case EXIT_INTR_WINDOW:
    /*
     * HW2: the guest can take an interrupt now.  Resume it; vmrun()
     * injects the pending interrupt on this entry.
     */
    /*
     * The CPU took this exit before the first instruction at which
     * RFLAGS.IF = 1 and there is no blocking by STI or MOV SS (SDM 28.2).
     * The saved RIP is that instruction, which has not run (SDM 30.3.3),
     * so RIP is not advanced.  The control itself stays set until
     * vmxinject() writes the base controls back on the next entry;
     * left on, it would exit before every instruction.  No trip to user
     * space: the kernel has everything needed to inject.
     */
    return 1;

This exit (reason 7) happens only while vmxinject() has turned on “interrupt-window exiting”. It means “the guest could take an interrupt now”. There is nothing to emulate and nothing to skip: SDM 30.3.3 says the saved RIP is “that which would be in the register had the VM exit not occurred”, the instruction that is about to run. Returning 1 loops back in vmrun(), whose next vmxinject() finds the window open and injects. Without this case, default sends the exit to runxv6, which stops with unhandled exit 7.

Step 5: Your turn: vmxinject()

SDM: 27.8.3 “VM-Entry Controls for Event Injection” (Table 27-18, the injected-event identification); 27.9.3 (Table 27-21, the original-event identification); 30.2.4 “Information for VM Exits During Event Delivery”; 30.2.5 (the instruction length); 29.6 “Event Injection”, 29.6.1.1 “Details of Event Injection”, 29.6.1.2 “VM Exits During Event Injection”; 28.2 and 29.7.5 “Interrupt-Window Exiting and Virtual-Interrupt Delivery”; 27.4.2 (Table 27-3); the entry checks in 29.2.1.3, 29.3.1.4 and 29.3.1.5; Vol. 3A 7.12.1.3 (an interrupt gate clears IF).

Goal. vmrun() calls vmxinject(vm) before every VM entry, with the VMCS current, host interrupts off, and GUEST_RFLAGS already written from vm->rflags. It decides two things for this entry:

1. which event, if any, VM entry injects (at most one per entry), written to the injected-event identification field;
2. whether the guest should exit as soon as it can take an interrupt, written as the “interrupt-window exiting” bit of the primary processor-based controls.

The fields and constants (all in vmx.h; FL_IF is in mmu.h):

The injected-event identification (Table 27-18) and the original-event identification (Table 27-21) have the same layout:

Other constants: INTERRUPTIBILITY_STI and INTERRUPTIBILITY_MOVSS (bits 0 and 1 of the interruptibility state), CPU_INTR_WINDOW_EXITING (bit 2 of the primary controls, Table 27-6), FL_IF (bit 9 of RFLAGS).

The rules, in priority order.

1. Re-inject an event whose delivery the last exit interrupted. If vm->idtvec has the valid bit, the exit happened while the CPU was delivering that event to the guest, and the event did not reach its handler (SDM 30.2.4; SDM 30.1: CS, RIP, RSP and RFLAGS were not updated for it). Inject it again:

   • the identification is the original one with bit 12 cleared (bit 12 is undefined in the original-event field but reserved in the injected one);
   • if bit 11 is set, also write the saved error code to the injected-event error-code field;
   • if the type is 4, 5 or 6 (a software interrupt or exception), also write the saved instruction length to the VM-entry instruction-length field.

   This comes first because this event happened first, and only one event can be injected per entry.

2. Otherwise, inject the highest pending vector, if the guest can take it now. “Can take it” means exactly what it means for a real CPU: RFLAGS.IF is 1 in vm->rflags, and vm->intrstate shows no blocking by STI or by MOV SS. Then the identification is the vector, type 0 (external interrupt) in bits 10:8, and the valid bit 31. No error code and no instruction length. Clear that vector’s bit in vm->pending: it has left the pending set now.

3. Ask for an interrupt-window exit whenever an interrupt is still pending after this decision. That covers two cases: an interrupt is pending but the guest cannot take one now (rule 2 did not inject), and this entry injects something but more interrupts wait behind it. Set CPU_INTR_WINDOW_EXITING in the value you write to the primary controls. The CPU then exits (reason 7) before the first instruction at which IF is 1 and nothing blocks interrupts (SDM 28.2), and the next entry injects. This is safe even when this entry injects an event: the window exit comes only after the injected event has been delivered (SDM 29.7.5, “These events occur after any event injection”), and delivery through an interrupt gate clears IF, so the window opens only when the guest’s handler returns with iretq.

Finally, write both fields on every entry: the identification (0 when nothing is injected) and the primary controls, computed from vm->cpuctl plus, when rule 3 says so, CPU_INTR_WINDOW_EXITING. Writing the controls every time is what turns window exiting off again once nothing is pending.

What is really lost without rule 1? Look at what each kind of event leaves behind when its delivery is interrupted:

• An interrupt we injected. Its bit in vm->pending was cleared when we injected it, and the guest never saw it. The original-event identification is now its only record. (Real hardware is in the same position: when the delivery of an external interrupt causes a VM exit, the interrupt controller has already been acknowledged and the interrupt is no longer pending, SDM 30.1.) Drop it, and a timer tick is lost, or worse, a disk interrupt: the process in iderw() sleeps forever.
• A guest exception, such as a page fault, or an INT $64 system call. Not lost: the guest’s RIP still points at the instruction that raised it, so when the guest resumes, the instruction runs again and raises the event again. Re-injecting is still the right thing (it does not run the instruction twice), but then a software interrupt needs its instruction length: injection pushes the current RIP plus the VM-entry instruction length as the return address (SDM 29.6.1.1). With length 0, the handler’s iretq returns to the INT itself, and the guest makes the system call twice.

When can such an exit happen here? SDM 30.2.4 lists the cases: an exception that exits because of the exception bitmap, a task switch, an APIC-access exit, or an EPT violation or misconfiguration during delivery. Our exception bitmap is 0, xv6 uses no task gates, and all of guest memory is mapped from the start, so in this hypervisor rule 1 almost never fires. A hypervisor that sets exception-bitmap bits, or that maps guest memory lazily on EPT violations, as KVM does, meets it all the time. Write it anyway: it is the rule that keeps the hypervisor correct after you change something else.

Hints.

• Start from two local values: the controls vm->cpuctl and an identification of 0. Change them as the rules say, and write them both at the end.
• nextpending() (Step 2) gives you the vector for rule 2, and calling it again after rule 2 tells you whether rule 3 applies.
• Use vm->rflags, not a vmread() of GUEST_RFLAGS: they are the same value here, and vm->rflags is clearer. Use vm->intrstate; the interruptibility state is never written back to the VMCS, it simply stays there from the last exit.
• Once rule 1 has used vm->idtvec, set it to 0. (vmrun() overwrites it after the next exit anyway; clearing it makes the “consumed” explicit.)
• Never OR CPU_INTR_WINDOW_EXITING into vm->cpuctl itself: then it can never be turned off.
• Rule 2’s check is not a courtesy. VM entry itself refuses an injected external interrupt when RFLAGS.IF is 0 (SDM 29.3.1.4) or when the interruptibility state has bit 0 or 1 set (SDM 29.3.1.5): the entry fails with exit reason 33 (invalid guest state).
• The injection fields are checked too (SDM 29.2.1.3): reserved bits 30:14 and 12 must be 0, the type must not be 1 or 7 (7 only for special cases), bit 11 must be 0 for anything but a hardware exception that has an error code, and the instruction length must be at most 15 for types 4 to 6. A violation makes VMLAUNCH fail with VM-instruction error 7 (invalid control fields, Table 33-1). Every VM exit clears the valid bit of the injected-event identification (SDM 27.8.3), so a stale value is never re-injected by accident; still, write 0 explicitly when there is nothing to inject.
• vmxinject() must not call cprintf() in its final version, but a temporary one is the best test (see the Checkpoint).

How to check. See the Checkpoint: with a temporary cprintf you should see vector 32 injected about once per host tick and vector 46 after the guest’s first disk command.

Step 6: The guest’s timer in runxv6

Reference: xv6’s trap.c (IRQ_TIMER) and lapic.c; no SDM.

Given. The guest’s local APIC timer is never going to fire (it is a page of memory), so runxv6 plays its part. Before its main loop it records the host’s tick count with uptime():

  /* uptime() counts host timer ticks; see the end of the loop. */
  lasttick = uptime();

In the loop, an EXIT_EXTERNAL_INTR that reaches user space (Step 4a) has nothing to emulate:

    case EXIT_EXTERNAL_INTR:
      /* The host timer ticked: just a chance to run below. */
      break;

and after every exit, whatever its reason, the end of the loop checks the clock:

    /* Once per host tick: the guest's timer interrupt. */
    /*
     * The guest's local APIC timer is not emulated (its registers are a
     * plain page), so we play its part: one vector 32 per host tick.
     * The guest's trap() counts ticks on it and preempts the running
     * process, as it would at each local APIC timer interrupt, and its
     * lapiceoi() just writes the plain page. If the guest has not taken
     * the previous one yet, the two merge into one pending interrupt.
     */
    if(uptime() != lasttick){
      lasttick = uptime();
      vminject(vm, VEC_TIMER);
    }

VEC_TIMER is 32 + 0, T_IRQ0 + IRQ_TIMER in xv6’s traps.h. Because vmrun() returns at least once per host tick (Step 4a), this runs at least once per tick, so the guest gets one timer interrupt per host tick, or fewer when runxv6 waits for the CPU and ticks merge. The other two vminject() calls are in the device emulation, which Part 5 covers: in rxpush() and in the IER write of com1() (vector 36), and in ide() and idedone() (vector 46).

One timer tick, end to end

Here is everything that happens for one guest timer interrupt, in the common case where the guest happens to have interrupts off when the tick comes. Say the guest is in its scheduler loop, scanning the process table while it holds ptable.lock, so its RFLAGS.IF is 0.

1. Host timer. The host CPU’s local APIC timer fires while the guest runs. External-interrupt exiting causes a VM exit, reason 1. The interrupt is not acknowledged; it stays pending in the host’s local APIC (SDM 30.1). The exit leaves host interrupts off.
2. Read and clear. vmrun() reads the exit fields, including intrstate and idtvec (Step 3c), VMCLEARs the VMCS and calls popcli().
3. The host takes its interrupt. With interrupts on, the pending timer interrupt goes through the host’s IDT to trap(); on CPU 0 it increments ticks. (The host may also reschedule runxv6 here.)
4. Back to user space. vmxhandle() sees ticks != vm->starttick and returns 0; vmrun() returns the exit to runxv6.
5. runxv6 injects. Its switch does nothing for EXIT_EXTERNAL_INTR; then uptime() != lasttick, so it calls vminject(vm, 32), which sets bit 32 of vm->pending. It polls the console and calls vmrun() again.
6. Entry with window exiting. vmrun() sets starttick = ticks, loads the VMCS and calls vmxinject(). No interrupted event; vector 32 is pending; but the guest’s RFLAGS.IF is 0, so nothing is injected and CPU_INTR_WINDOW_EXITING is turned on. VMLAUNCH.
7. The guest opens the window. The guest finishes its scan and calls release(&ptable.lock), whose popcli() executes sti. STI blocking holds interrupts off for one more instruction; before the instruction after that, IF is 1 and nothing blocks, so the CPU exits with reason 7 (EXIT_INTR_WINDOW). The saved RIP is the instruction that has not run yet.
8. Inject. vmxhandle() returns 1, without a trip to user space. vmrun() loops, and vmxinject() now finds IF = 1 and no blocking: it writes 0x80000020 (valid, type 0, vector 32) to the injected-event identification, clears the pending bit, and, with nothing else pending, writes the controls without window exiting. VMLAUNCH.
9. Delivery. As the last step of VM entry, the CPU delivers vector 32 through the guest’s IDT (SDM 29.6.1), exactly as if it had come from a local APIC: it pushes the guest’s SS, RSP, RFLAGS, CS and the RIP saved in step 7, and, since xv6’s vector 32 is an interrupt gate, clears IF. The guest runs vector32, alltraps and trap().
10. The guest’s handler. trap() matches T_IRQ0 + IRQ_TIMER: cpuid() is 0, so it increments the guest’s ticks and wakes up sleepers. lapiceoi() writes the EOI register, which is a store to the zeroed page: no exit, and nobody notices. If a process was running, trap() calls yield(). Finally iretq restores IF = 1, and the guest goes on until the next exit.

If the guest already had IF = 1 at step 6, steps 7 and 8 disappear: the first entry injects at once.

Checkpoint

Build and boot as in Part 3 (make, make fs.img, then make qemu-nox on CADE or make bochs-nox elsewhere). The guest still cannot boot all the way: its first disk read uses rep insl, string I/O that your kernel handles only after Part 5. But you can watch the interrupts go in.

Add two temporary cprintfs. One goes at the top of vminject(), after its checks, and shows each request from runxv6:

  cprintf("vminject %d\n", vector);

The other goes in vmxinject(), just before the final two vmwrite()s, and shows each injection, whenever this entry injects something:

  if(info & INTR_INFO_VALID)
    cprintf("inject 0x%x\n", (uint64)info);

(use the name of your own variable; %x reads 64 bits, so cast). Then run runxv6 xv6kernel xv6fs.img. You should see something like this (the vminject 32 lines are interleaved with the guest’s boot messages, and their number depends on how fast your machine boots the guest):

$ runxv6 xv6kernel xv6fs.img
runxv6: booting xv6kernel (type Ctrl-] to stop)
xv6...
vminject 32
cpu0: starting 0
cpu0: no VMX support
vminject 32
inject 0x80000020
vminject 46
inject 0x8000002e
vminject 32
vminject 32
vminject 32
...

What to check:

1. vminject 32 comes about once per host tick, for the whole run, including after the guest has stopped making progress: runxv6 keeps running and keeps playing the timer.
2. The first inject 0x80000020 (valid, type 0, vector 32) comes only after cpu0: no VMX support. The guest boots with IF = 0, so the ticks requested during boot merge into one pending vector 32, which waits, with window exiting on, until the guest’s scheduler executes its first sti. (On a very fast machine no tick may fall in that window; then the first inject line is the disk’s.)
3. vminject 46 comes right after the guest’s first IDE read command, and inject 0x8000002e (vector 46) shortly after it, once the guest has gone to sleep in iderw(), released its locks, and enabled interrupts.
4. After that there are no more inject lines and no sb: size 2000 ... line. The guest’s IDE handler executes rep insl, string I/O that your HW1 vmxhandle() does not handle, so runxv6 gets the same I/O exit over and over, while the guest sits with IF = 0 inside its handler. The timer requests pile up as one pending vector. That is expected until Part 5.
5. Type Ctrl-]: runxv6 must stop and return to the host’s shell.

Remove both cprintfs when you are done; at one line per host tick they swamp the console.

The once-per-tick return of Step 4a cannot really be seen yet, because this guest stops before it ever idles. You will see it in Part 5: at the guest’s shell prompt the guest only spins in its scheduler, and without Step 4a nothing would ever bring runxv6 back to deliver your keys.

If something fails:

Questions

Answer these in answers-hw2.md, two or three sentences each.

4.1. nextpending() delivers the highest pending vector first. Which rule of a local APIC does this imitate (SDM 13.8.3), and what does a real local APIC also take into account that we ignore? With the timer (32) and the disk (46) both pending, which goes first, and what happens to a second timer tick that arrives before the guest has taken the first? 4.2. Rule 1 of vmxinject() re-injects an event whose delivery was interrupted by a VM exit. For each of (a) a timer interrupt we injected, (b) a page fault raised by a guest instruction, (c) an int $64 system call: is the event lost if we do not re-inject it, and why? For (c), what goes wrong if we re-inject it but forget the instruction length? 4.3. Why must vmrun() return to user space once per host tick? Describe what the guest would do, and what you would see at the terminal, if it returned only on I/O and HLT exits while the guest sits at an idle shell prompt and you type a key. Why not return on every external-interrupt exit instead? 4.4. Why does vmxinject() not inject an external interrupt when the guest has RFLAGS.IF = 0? Give two reasons: one from the VM-entry checks, one from what xv6 is doing when IF is 0. Why is waiting for the next exit of any kind not good enough, so that we need interrupt-window exiting? 4.5. Our guest has one CPU. List what the hypervisor and runxv6 would need in order to give it two. Think about the local APIC ID each CPU reads from the same guest-physical page, how xv6 starts the second CPU (lapicstartap()), where each interrupt should go, the VMCS, and the fact that an xv6 process has a single thread.

Part 5: Devices and string I/O

Parts 1 to 4 gave the guest memory, firmware tables and interrupts. What it still lacks is hardware: a console to print to and read from, and a disk that holds its file system. In this part you make the guest’s disk work. When you finish, the whole homework works end to end: runxv6 xv6kernel xv6fs.img boots xv6 inside your hypervisor to a shell prompt, and you can type commands at it.

Almost all the device code is in runxv6.c and is given to you. You add one kernel step, so that the kernel decodes string I/O (INS and OUTS, which xv6’s disk driver uses to move whole sectors), and you write one function in runxv6, idestring(), that carries out such a transfer.

Which devices does the guest have

The guest believes it runs on a PC. This table lists everything of that PC that xv6 touches, and who plays each part.

Any other port reads as all ones, and writes to it are ignored, as on a real bus where no device answers.

Are we ignoring the guest’s screen? Yes, and nothing is lost. xv6’s consputc() sends every character to both cgaputc() (the screen) and uartputc() (the serial port). runxv6 shows the serial copy: this is the same thing make qemu-nox does for the host xv6. The screen’s characters land in ordinary guest RAM at 0xB8000, and only the CRT controller’s cursor registers need emulating, so that cgaputc() does not panic (Step 6).

Does struct vmexit describe the devices? No. For port I/O it says what the CPU trapped: the port, the size, the direction, the value of an OUT, and (new in this part) four fields for string instructions. It never says “this was the disk”. runxv6’s doio() decides that, with a switch on the port number. This is the KVM and QEMU split again: KVM reports KVM_EXIT_IO with a port number, and QEMU knows which device model lives at that port.

What is given and what you write

The given device code is still worth reading closely: Steps 3 to 7 walk through it, because idestring() has to fit into it.

Step 1: Decode string I/O in the kernel

SDM: 30.2.1 “Basic VM-Exit Information” (Table 30-5 “Exit Qualification for I/O Instructions”, and the item “Guest linear address”); 27.9.1 (the guest-linear address field among the VM-exit information fields); Appendix B.4.2, Table B-13 (its encoding, 0x640A); 28.1 (I/O exits are fault-like); 30.2.5 (the VM-exit instruction length covers INS and OUTS).

In HW1 your EXIT_IO case gave up on string instructions with a line like this:

    if(e->qual & (1 << 4))
      return 0;  /* string I/O (INS/OUTS) is not supported */

Remove it, and put this in its place, at the top of the EXIT_IO case, after the comment about the exit qualification and before the line that sets e->io.port:

    /*
     * HW2: string I/O (INS/OUTS, as in xv6's insl/outsl for the IDE
     * disk).  Bit 5 says it has a REP prefix.  vmrun() has already read
     * the guest-virtual address of the memory operand into e->io.addr.
     * runxv6 does the whole transfer at once (all RCX iterations of a
     * REP) and updates RCX and RSI or RDI, so the RIP can be advanced
     * here, as for any other I/O exit.
     */
    /*
     * Why that is right: the exit is fault-like (SDM 28.1), so no
     * iteration has run yet and RCX still holds the full count.  Once
     * runxv6 has done all of them and set RCX to 0 and RSI or RDI past
     * the buffer, the guest is in the state the CPU leaves at the end of
     * the instruction, and the next one must run.  A monitor that did
     * only part of the work would instead leave RIP alone: the CPU
     * would run the REP instruction again (exiting again) for the
     * iterations that remain.
     */
    if(e->qual & (1 << 4)){
      /* Bit 4 of the exit qualification: INS or OUTS (Table 30-5). */
      e->io.string = 1;
      e->io.rep = (e->qual >> 5) & 1;
      /*
       * A REP prefix repeats the transfer RCX times (xv6's insl and
       * outsl load RCX with the count); without one, a single unit
       * moves.  RCX is the guest's value, saved by vmx_exit.
       */
      e->io.count = e->io.rep ? vm->regs.rcx : 1;
    }

Table 30-5 gives the two bits we need: bit 4 is 1 for a string instruction, and bit 5 is 1 if it has a REP prefix. The rest of the qualification (size in bits 2:0, direction in bit 3, port in bits 31:16) means the same for INS/OUTS as for IN/OUT, so your HW1 decoding below this block stays as it is. In particular, for rep insl the size field is 3 and your (e->qual & 7) + 1 gives 4.

The count is not in the qualification. A REP instruction repeats RCX times, and RCX is a general-purpose register, which vmx_exit saved in vm->regs. Without REP, the instruction moves one unit.

Then, further down in the same case, change the condition on the line that records the data of an OUT:

    /*
     * HW2: not for OUTS, whose data is in guest memory at e->io.addr;
     * runxv6 reads it there with vmmem().
     */
    if(!e->io.in && !e->io.string)
      e->io.data = vm->regs.rax & iomask(e->io.size);

OUTS does not send RAX: it sends bytes from memory at DS:RSI. RAX holds something unrelated, and io.data stays 0. Leave the vm->rip += inslen and the return 0 that follow as they are: the RIP is advanced for string I/O too. The instruction length includes the REP prefix (rep insl is the two bytes F3 6D).

The second edit is in vmrun(). The kernel must tell runxv6 where in guest memory the data goes or comes from. The CPU computes that address for us and stores it in the VMCS field “guest-linear address”. Like every VMCS field, it can only be read while the VMCS is current, so the read goes with the others, before vmclear(). Add these lines at the end of the else branch that reads the exit information, after the reads you added in Part 4 and before the brace that closes the branch:

      if(vm->idtvec & INTR_INFO_VALID){
        ...                              /* from Part 4 */
      }

      /*
       * For INS and OUTS (bit 4 of the qualification), the CPU saves the
       * linear address of the memory operand: the ES base plus RDI for
       * INS, the DS (or override) base plus RSI for OUTS (SDM 30.2.1).
       * xv6 runs in 64-bit mode, where those bases are 0, so it is RDI
       * or RSI itself.  For other exits the field is not meaningful.
       */
      if(e.reason == EXIT_IO && (e.qual & (1 << 4)))
        e.io.addr = vmread(GUEST_LINEAR_ADDRESS);

    }
    /*
     * Clear the VMCS so no CPU caches it: once interrupts are back on,
     * the timer may move this process to another CPU.  This must happen
     * before popcli(), and on every path out of the entry.
     */
    vmclear(V2P(vm->vmcs));

Why not simply use the guest’s RDI or RSI from vm->regs? In 64-bit mode the answer would be the same, but the VMCS field is the general one: it adds the segment base, it honors a segment override prefix on OUTS, and it clears bits 63:32 if the guest was not in 64-bit mode (SDM 30.2.1). It is a linear (guest-virtual) address, translated by the guest’s page tables, not a guest-physical one. That is why runxv6 will pass it to vmmem() with VMMEM_VIRT, which walks the guest’s page tables with your gva2gpa() from Part 2.

vmrun() clears e at the top of every pass, so for all other exits the four new fields are 0.

Step 2: What the REP prefix means for the hypervisor

SDM Vol. 2: the INS/INSB/INSW/INSD, OUTS/OUTSB/OUTSW/OUTSD and REP/REPE/REPZ/REPNE/REPNZ instruction pages. Vol. 1: the EFLAGS register (DF, bit 10). Vol. 3: 28.1 (fault-like exits), 28.1.3 (I/O instructions).

This is xv6’s insl(), in x86.h:

static inline void
insl(int port, void *addr, int cnt)
{
    asm volatile("cld; rep insl" :
            "=D" (addr), "=c" (cnt) :
            "d" (port), "0" (addr), "1" (cnt) :
            "memory", "cc");
}

ideintr() calls it as insl(0x1f0, b->data, BSIZE/4). So RDX = 0x1F0, RDI = b->data, and RCX = 128. cld clears the direction flag (DF), and then rep insl does this, until RCX is 0:

1. read 4 bytes from port DX and store them at ES:RDI;
2. RDI += 4 (it would be RDI -= 4 if DF were 1);
3. RCX -= 1.

That is 128 iterations, 512 bytes, one sector, in one instruction. outsl() is the mirror image: cld; rep outsl loads 4 bytes from DS:RSI, writes them to port DX, and advances RSI.

Our VMCS sets “unconditional I/O exiting” (HW1), so every INS and OUTS exits. The exit is fault-like (SDM 28.1): the instruction has not executed at all, not even the first iteration. At the exit, RCX still holds the full count, 128, and RDI (or RSI) still points to the start of the buffer. (You can check this with a temporary printf of getreg(VMREG_RCX) in idestring(): it prints 128 every time.)

There are two ways to emulate it:

• One iteration per exit. Move 4 bytes, update RDI and RCX, and leave RIP pointing at the instruction. The guest re-executes rep insl, which exits again, 128 times per sector. Simple, but each exit is a round trip through the kernel and into runxv6.
• All iterations in one exit. This is what we do. The kernel reports the guest-linear address of the buffer and the count, and advances RIP past the instruction (Step 1). runxv6 moves all count * size bytes at once and then leaves the registers exactly as the CPU would have at the end of the instruction: RDI (for INS) or RSI (for OUTS) advanced by the number of bytes, and RCX = 0 if there was a REP prefix. One exit per sector instead of 128.

Doing all the iterations at once rests on four assumptions, all true for xv6:

• DF = 0, so addresses go up. insl() and outsl() execute cld first.
• The whole buffer is mapped. On hardware, a buffer that runs into an unmapped page would fault partway, with RCX and RDI showing the progress so far. We do not model that: runxv6 stops. xv6’s buffers are in its kernel memory, always mapped.
• 64-bit address size, so the registers are RCX, RDI and RSI, not ECX, EDI and ESI. xv6-64 runs its driver in 64-bit mode with no address-size prefix.
• The count fits the transfer the drive expects: idestring() checks it against what the command asked for.

Step 3: The dispatch, doio() (given)

SDM: Table 30-5 (the kernel has decoded it into e->io).

/*
 * Called for every EXIT_IO, after the kernel has decoded the exit
 * qualification (SDM 30.2.1, Table 30-5: size, direction, string, REP
 * and port number) into e->io and moved the guest's RIP past the
 * instruction. Choose the device by port number.
 */
static void
doio(struct vmexit *e)
{
  uint p = e->io.port;

  if(p >= COM1 && p < COM1 + 8){
    com1(e);
  } else if((p >= 0x1F0 && p < 0x1F8) || p == 0x3F6){
    ide(e);
  } else if(p == 0x3D4 || p == 0x3D5){
    crt(e);
  } else if(e->io.string){
    /*
     * A string instruction would also need memory, RCX, and RSI or RDI
     * updated; rather than fake that for a device we do not know, stop.
     */
    die("string I/O to an unknown port");
  } else if(e->io.in){
    /*
     * No device: the PICs (0x20, 0x21, 0xA0, 0xA1), which xv6 only
     * masks; the keyboard (0x60, 0x64), which never interrupts here; the
     * CMOS clock (0x70, 0x71). Reads float high.
     *
     * On a PC, a read from a port that no device answers returns all
     * ones; completein() keeps the low size bytes. An OUT to such a
     * port is simply dropped, as on a real bus.
     */
    completein(e->io.size, 0xFFFFFFFF);
  }
}

This is runvm’s doio() from HW1 grown into a small bus: each range of ports belongs to one device function. completein() is the runxv6 version of the function you wrote in HW1 Part 6.

Note the string case for unknown ports. A plain IN from nowhere is easy to fake (all ones in RAX), but a string instruction would also have to fill memory and update RCX and RDI, and no guest has a reason to do that to a port with no device. Stopping makes such a bug visible at once.

Step 4: The serial port, COM1 (given)

Reference: the PC16550D UART datasheet (National Semiconductor, now TI), for the registers and their bits; xv6’s uart.c for how they are used.

COM1 is eight byte-wide registers at ports 0x3F8 to 0x3FF. com1() models the ones xv6 uses:

The transmit side is HW1’s serial port: an OUT to offset 0 goes to our standard output, and the line status always says “transmitter empty”, so uartputc() never waits.

The receive side is new. Keys go into a 256-character queue, and the guest is interrupted:

static void
rxpush(char c)
{
  if(rxw - rxr < sizeof(rxbuf))
    rxbuf[rxw++ % sizeof(rxbuf)] = c;
  if(ier & 0x01)
    vminject(vm, VEC_COM1);
}

vminject(vm, 36) asks the kernel to deliver vector 36 (IRQ 4) at the next VM entry at which the guest can take it (Part 4). The guest’s trap() calls uartintr(), which calls consoleintr(uartgetc). uartgetc() reads the LSR, sees bit 0 (data ready), and reads the character from RBR, which removes it from the queue. It repeats until the LSR says the queue is empty, so one interrupt drains every key that has arrived. The interrupt is only raised if the guest has enabled it (IER bit 0); com1() also raises it when the guest enables it while keys are already waiting, as a 16550 would.

How does the guest’s keyboard work? It does not: the guest’s PS/2 keyboard never interrupts (IRQ 1 is never injected) and its ports are not emulated, so kbdintr() never runs. Your keys reach the guest through its serial port instead. xv6 has two input paths, and both end in consoleintr():

• the PS/2 keyboard: IRQ 1, kbdintr(), which reads scancodes from ports 0x60 and 0x64 and translates them to characters;
• the serial port: IRQ 4, uartintr(), which reads characters from COM1.

runxv6 gets characters from the host’s console, not scancodes. And the host xv6 itself gets your keys through its serial port: under make qemu-nox or make bochs-nox the terminal is connected to the emulated PC’s COM1. So the serial path is the natural one, with no translation in either direction. Here is the whole path of one key:

your terminal
  -> QEMU or Bochs: the host PC's COM1 receives the byte, IRQ 4
  -> host xv6: uartintr() -> consoleintr()  (raw mode: no echo, no editing)
  -> runxv6: pollconsole(): fdready(0), read(0)
  -> runxv6: rxpush(): queue it, vminject(vm, 36)
  -> kernel: vmxinject() delivers vector 36 at the next VM entry
  -> guest xv6: uartintr() -> consoleintr()  (echo, line editing)
  -> guest sh: read() returns the line

The echo you see on the screen comes from the guest: its consoleintr() echoes each key with consputc(), which goes out through the guest’s COM1 and comes back to your terminal through runxv6’s write(1, ...).

Step 5: Reading the console without stopping the guest, pollconsole() (given)

Reference: the fdready() and consraw() system calls (Part 1; sysfile.c, console.c).

/*
 * Read the keys typed on our console, if any, without blocking.
 *
 * xv6 has no non-blocking read, so fdready() asks first whether a
 * read() would return at once. The console is in raw mode (consraw), so
 * read() returns each key as soon as it is typed, without waiting for a
 * whole line.
 */
static void
pollconsole(void)
{
  char buf[64];
  int i, n;

  while(fdready(0) > 0){
    if((n = read(0, buf, sizeof(buf))) <= 0)
      return;
    for(i = 0; i < n; i++){
      /* Our own key: never passed to the guest. */
      if(buf[i] == QUITKEY){
        consraw(0);
        printf(1, "\nrunxv6: stopped (Ctrl-])\n");
        exit();
      }
      /* Everything else arrives at the guest's COM1 receiver. */
      rxpush(buf[i]);
    }
  }
}

runxv6 is one process with one thread, and that thread spends its life in vmrun(). A plain read(0, ...) would sleep until you type, and the guest would stop running while it slept. xv6 has no O_NONBLOCK, select() or poll(), so Part 1 added fdready(fd), which says whether a read() would return at once. pollconsole() reads only then.

Three more details:

• Raw mode. main() calls consraw(1) before the run loop. In raw mode the host console neither echoes nor collects lines: each key is available to read() as soon as it arrives. Without it, every key would be echoed twice (once by the host, once by the guest), and the guest would get nothing until you pressed Enter.
• When it runs. At the bottom of runxv6’s loop, after every return from vmrun(). The kernel returns to user space at least once per host tick (Part 4), even if the guest does no I/O, so a key waits at most one tick before it reaches the guest.
• Ctrl-] is ours. It is ASCII 0x1D, which the guest has no use for. pollconsole() keeps it from the guest, turns raw mode off (otherwise the host shell would get a raw console), and exits; exit() frees the VM. Every other key, Ctrl-C and Ctrl-P included, goes to the guest.

Step 6: The CRT controller (given)

Reference: the Motorola 6845 CRT controller (registers 14 and 15, the cursor address); xv6’s cgaputc() in console.c.

static uchar crtindex, crtregs[256];  /* any uchar index is in range */

static void
crt(struct vmexit *e)
{
  if(e->io.port == 0x3D4){
    /* The index port: which register 0x3D5 refers to. */
    if(e->io.in)
      completein(e->io.size, crtindex);
    else
      crtindex = e->io.data;
  } else {
    /* The data port: the selected register. */
    if(e->io.in)
      completein(e->io.size, crtregs[crtindex]);
    else
      crtregs[crtindex] = e->io.data;
  }
}

Nobody looks at the guest’s screen, so why emulate any of it? Because of how cgaputc() finds the cursor. It writes 14 to port 0x3D4 and reads the high byte of the position from 0x3D5, then does the same with register 15 for the low byte. The position is 80 * row + column. Then it checks it:

  if(pos < 0 || pos > 25*80)
    panic("pos under/overflow");

With no device at 0x3D5, both reads return 0xFF, the position is 0xFFFF, and cgaputc() panics. Now look at panic(): it prints "lapicid %d: panic: " with cprintf(), which calls consputc(), which calls uartputc('l') (you see an l) and then cgaputc('l'), which reads the cursor again, gets 0xFFFF again, and panics again. Each nested panic prints one more l, until the kernel stack overflows and the guest triple-faults. The first console line goes through this path (the banner xv6... does not: uartinit() prints it with uartputc() alone), so right after the banner you see the first letter of cpu0: starting 0, then a row of lllllll, then runxv6: guest triple fault.

This really happened while the reference solution was being built, before crt() existed. The fix is the smallest device that satisfies cgaputc(): registers that read back what was written, starting at 0 (the top-left corner). The characters themselves go to 0xB8000, which is ordinary guest RAM.

Step 7: The IDE disk registers (given)

Reference: the ATA/ATAPI command set and its PIO data-in and data-out protocols (we do not cite section numbers: they differ between the ATA revisions); xv6’s ide.c.

The primary IDE channel’s registers are at 0x1F0 to 0x1F7: data (0), sector count (2), the LBA bytes (3, 4, 5), drive/head (6), and status/command (7). Port 0x3F6 is the alternate status (read) and device control (write). The comment above idecommand() in runxv6.c lists them all.

Status. Four bits matter. BSY (0x80): the drive is busy, the other bits are not valid. DRDY (0x40): ready for a command. DRQ (0x08): ready to move data through the data port. ERR (0x01): the last command failed. Our medium is memory, so the drive is never busy.

idestatus starts out as DRDY, and that is not a detail. ideinit() first spins in idewait() until the status has BSY clear and DRDY set. An absent device, reading 0xFF, has BSY set, so the guest would spin forever. Then ideinit() selects drive 1 and decides the disk is present if the status is nonzero. Our status does not depend on the drive selected, so it is.

A command. idestart() writes the sector count, the three LBA bytes and drive/head, then the command. idecommand() reassembles the LBA28 sector number (bits 7:0, 15:8 and 23:16 from ports 0x1F3 to 0x1F5, bits 27:24 from the low four bits of drive/head) and checks it:

• a command for drive 0, for more sectors than idebuf holds (8), or past the end of the image fails with ERR. Only drive 1 exists: the guest’s root file system is on disk 1 (ROOTDEV), and nothing loads the guest kernel from disk 0;
• READ SECTORS (0x20) or READ MULTIPLE (0xC4): copy the sectors from the image into idebuf, set DRQ, and inject vector 46 at once;
• WRITE SECTORS (0x30) or WRITE MULTIPLE (0xC5): set DRQ and wait for the data. No interrupt yet.

On a real drive, READ SECTORS has a data phase and an interrupt per sector, and READ MULTIPLE per group of sectors. ideintr() reads a whole block with one insl() after one interrupt, so both are treated alike: all the sectors are ready at once. (With BSIZE = 512, xv6 always sends READ SECTORS and WRITE SECTORS anyway.)

Interrupt timing. For a read, the interrupt comes when the command is written, since the data is ready immediately. iderw() holds idelock at that moment, so the guest has interrupts off; the kernel keeps vector 46 pending and delivers it as soon as the guest sets IF again (Part 4). For a write, the interrupt comes when the last byte has arrived, in idedone():

/* n more bytes of the transfer went through the data port. */
static void
idedone(int n)
{
  idepos += n;
  if(idepos < idelen)
    return;
  /* The last byte moved: clear DRQ, as no more data is expected. */
  idestatus = IDE_DRDY;
  /*
   * A read is complete now; its interrupt came at the start. A write
   * has all its data: "write it to the medium" and raise the
   * completion interrupt. ideintr() sees B_DIRTY, skips reading, and
   * wakes the process sleeping in iderw().
   */
  if(idewriting){
    memmove(disk + idesector * 512, idebuf, idelen);
    vminject(vm, VEC_IDE);  /* the write is done */
  }
}

idepos and idelen are the position in, and the size of, the current transfer in idebuf. Every path that moves data through the data port must end with idedone(): the single inl/outl cases in ide() do, and so must your idestring(). If a write never reaches idedone(), it is never “written” and never interrupts, and the guest process that started it sleeps forever.

Writes stay in memory. idedone() writes into disk, runxv6’s copy of xv6fs.img, never into the file. Writing one block back would mean rewriting the whole image, since xv6 has no lseek(). So every run of runxv6 starts from the same clean disk; a file you create in the guest is gone after Ctrl-].

Step 8: Your turn: idestring()

SDM: 30.2.1 (the guest-linear address); 28.1 (fault-like exits). SDM Vol. 2: the INS and OUTS instruction pages and the REP prefix. The vmmem() comment in vmxapi.h for the VMMEM_* flags.

The stub in your tree is

static void
idestring(struct vmexit *e)
{
  die("idestring not written");
}

ide() calls it for every string instruction on the data port 0x1F0 (any other IDE port makes ide() stop). That is rep insl in ideintr(), reading a block, and rep outsl in idestart(), writing one.

Goal. Carry out the whole string instruction in one call, as Step 2 describes: move all the bytes between the guest’s buffer and idebuf, and leave the guest’s registers as the CPU would have left them at the end of the instruction. The kernel has already advanced RIP past it (Step 1).

The contract.

• e->io.in says the direction: INS moves data from the drive into the guest’s memory, OUTS from the guest’s memory into the drive.
• e->io.count units of e->io.size bytes move, starting at the guest address e->io.addr, and at offset idepos of idebuf.
• Afterwards: RDI (for INS) or RSI (for OUTS) has moved forward past the buffer; RCX is 0 if the instruction had a REP prefix, and unchanged if it did not; and the drive knows the bytes went through.

Hints.

• The number of bytes is count * size. For xv6 it is always 512.
• Before you copy, check that the transfer fits in what the command asked for: compare with idelen - idepos. More than that is a guest bug, and copying it would run past the end of idebuf. Stop with die().
• Copy with vmmem(). The address is guest-virtual (it is b->data, a guest kernel address), so the flags must include VMMEM_VIRT. Think about which direction needs VMMEM_WRITE: it means “into the guest”. vmmem() returns -1 if any page is not mapped; stop with a message that says which instruction failed.
• Update RDI or RSI with getreg() (a wrapper around vmgetreg()) and vmsetreg(). DF is 0 (Step 2), so the address goes up.
• Set RCX to 0 only when e->io.rep is set.
• Finish with idedone(), giving it the number of bytes. It clears DRQ when the transfer is complete, and for a write it updates the disk and raises the completion interrupt.
• The single inl/outl cases in ide() (offset 0) do the same job one word at a time. Read them: your function is their string version.

How to check. With the stub, the guest stops at its first disk read, just after cpu0: no VMX support, with runxv6: idestring not written. With INS done but not OUTS, it reads its superblock, prints the sb: line, and then stops at its first write, which the log does at boot. With both, it boots to the shell: see the Checkpoint.

Checkpoint

This is the checkpoint for the whole homework. Build and boot the host as in HW1 (make qemu-nox on a CADE machine, or make bochs-nox), then run the guest at the host’s shell:

$ runxv6 xv6kernel xv6fs.img
runxv6: booting xv6kernel (type Ctrl-] to stop)
xv6...
cpu0: starting 0
cpu0: no VMX support
sb: size 2000 nblocks 1941 ninodes 200 nlog 30 logstart 2 inodestart 32 bmap start 58
init: starting sh
$

Everything after the first line comes from the guest. Now use it. Typing must work, with backspace, and each of these must do what it does on the host:

$ ls
$ cat README
$ echo hi > f
$ cat f
hi
$ mkdir d
$ ls d
$ stressfs
$ forktest
$ timertest
timertest: slept 50 ticks
timertest: preemption ok
$ usertests2
...
ALL TESTS PASSED

stressfs and usertests2 exercise the disk hard, writes included. timertest checks that the timer you inject in Part 4 ticks and preempts. Finally type Ctrl-]: runxv6 prints runxv6: stopped (Ctrl-]) and you are back at the host’s $ prompt, with a normal (not raw) console.

usertests1 fails in its sbrk test, which asks for about 100 MB of memory; the guest has 32 MB. That is expected. (usertests2 refuses to run twice on the same disk, but since every run of runxv6 starts from a fresh copy of xv6fs.img, you can run it again after restarting the guest.)

If something fails:

A note on RCX: leaving it at 128 after a REP transfer changes nothing you can see with today’s xv6, because insl() and outsl() ignore their outputs. But it is not what the CPU does, and a guest that looked at RCX afterwards (to count how much was transferred, for example) would be wrong. Leave the registers exactly as the hardware would.

Questions

Answer these in answers-hw2.md, two or three sentences each.

5.1. With our design, reading one 512-byte sector costs one VM exit for the data instead of 128. Explain why the kernel can advance RIP past a rep insl that has not executed a single iteration. What would the kernel and runxv6 have to do differently if runxv6 moved only one dword per exit?

5.2. Suppose a guest executed std before rep insl (DF = 1). What would the CPU do with RDI, and which bytes of memory would it fill? What would our idestring() do instead? Sketch the change that would handle it (which register tells you DF, and how would the address and the RDI update change?).

5.3. The guest’s keys arrive as serial input, never as PS/2 keyboard interrupts. Why is the serial port the natural choice here? What would runxv6 need in order to emulate the keyboard instead (which ports, which vector, and what would it have to do to each character)?

5.4. Nobody displays the guest’s VGA memory at 0xB8000, and yet the guest’s output is complete. Why? And why, even so, do the CRT controller’s two ports have to be emulated?

5.5. struct vmexit has no field that names a device: there is nothing like “disk read” or “serial write”. Where does the knowledge that port 0x1F0 is a disk live, and why is it better there than in the kernel? Name the matching pieces in KVM and QEMU.

What to hand in

Upload these files to Gradescope:

• vmx.c, vmxasm.S and runxv6.c: your hypervisor (with your HW1 work, which HW2 builds on) and your monitor;
• answers-hw2.md: your answers to the questions at the end of every part, numbered by part and question (1.1, 2.3, …). The starter tree has a template with every question and a slot for each answer (answers-hw2.md; your HW1 answers stay in answers.md).

Submitting on Gradescope

Upload exactly four files: vmx.c, vmxasm.S, runxv6.c and answers-hw2.md. Do not upload the rest of your tree. The autograder uses only vmx.c, vmxasm.S and runxv6.c, and the hw2-assign version of every other file, so any change you make to another file (a header, the Makefile, the file system, and so on) is not part of your submission. Your code must work with all the other files unchanged.

Submit the files from your tree after you merged hw2-assign. A vmx.c from HW1 alone lacks the functions the HW2 starter code calls (vmmem, vminject, …), so it does not build, and every test fails.

The autograder builds your code and runs it under Bochs, the same way you test it: the host boots, runs runvm helloguest, then runxv6, and the guest runs a small CPUID check, ls, stressfs, timertest and usertests2. As in HW1, it grades each part on its own: for Part N, it takes only Part N’s functions from you and the reference solution for everything else, so a mistake in one part does not cost you the points of the others. Then it runs all of your code together, which is where the HW2 changes inside your HW1 functions (vmcsinit, vmcreate, vmxhandle, vmrun) are tested. It also checks that your tree builds with -Werror.

Grading takes longer than in HW1, since every test boots two kernels, one inside the other, in an emulator: expect up to half an hour. A test whose guest hangs (for example, waiting for an interrupt that never comes) stops after five minutes without output, so a broken submission can take longer. You see the result of every test when grading finishes, with the output of the failing guest program and the end of the console, and you can resubmit as often as you like before the deadline. Test under QEMU or Bochs first rather than using the autograder as your test loop. Your answers-hw2.md is graded by hand.

Before you submit, check that a clean tree builds with -Werror, that runvm helloguest still works (HW1), and that in a fresh runxv6:

$ runxv6 xv6kernel xv6fs.img
...
init: starting sh
$ timertest
timertest: slept 50 ticks
timertest: preemption ok
$ usertests2
...
ALL TESTS PASSED
$ stressfs
...

and that Ctrl-] brings you back to the host’s shell.

Where this goes next

The guest has one CPU and no real interrupt controller: its local APIC and I/O APIC are memory pages, interrupts arrive by injection, and its timer ticks at the host’s rate. A natural next step is a virtual local APIC: either by trapping the guest’s accesses to its page and emulating the registers, or with VT-x’s APIC virtualization, where the CPU handles most of the work (SDM chapter 32). With a real local APIC, the guest can start a second CPU, which means a second VMCS and a second vmrun loop, one per virtual CPU. Other directions: keeping the VMCS loaded across exits and using VMRESUME (as KVM does), VPID to keep the guest’s TLB entries, and making the guest’s disk writes persistent.