HW1: A VT-x hypervisor for xv6 (part 1, “Hello world”)

This is the first homework in a series in which you build a hypervisor for xv6-64, one that runs inside xv6 itself. In this first assignment the hypervisor runs a “hello world” guest. In the next one you will extend it to boot a full copy of xv6 as the guest.

In this homework you will work on a small Intel VT-x hypervisor inside xv6-64. Its structure follows KVM. A user program, runvm, plays the role of QEMU: it loads a guest image, sets the guest’s registers, and emulates the guest’s devices. The kernel (vmx.c) plays the role of KVM: it runs the guest and handles only the VM exits that it must. All other exits go back to runvm.

The guest is a small “hello world” program written in C (helloguest.c) that prints Hello from a VT-x guest through an emulated serial port. The goal of the whole series is to run xv6 itself as a guest of xv6.

Read this document first, then read the code. The source files are heavily commented. Reading order that works well: vmxapi.h, runvm.c, helloguestasm.S, sysvmx.c, vmx.c, vmxasm.S, and vmx.h as a reference for the constants.

How to use this handout

The work is split into six parts, in the order in which a hypervisor comes together. Each part has the same shape:

1. a short introduction that says what the part achieves, and which functions are given and which you write;
2. steps: blocks of real code, each with the sections of the Intel manual that describe the mechanism and an explanation of what it does and why. You paste the blocks, in order, into the body of the function that has been emptied out in your tree, and you read the explanation until you could have written the block yourself;
3. a few Your turn spots, where we give you the goal, the manual sections and hints, but not the code;
4. a checkpoint, something visible that tells you the part works, with a table of symptoms and likely causes;
5. questions, whose answers go in answers.md, a template that is already in your repo.

The Intel manual is the real reference. Section numbers in this handout are for the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026), written “SDM”. Section numbers change between editions, so search for the titles if yours differs.

Getting the code

Clone the assignment branch of the course repository and work on it:

$ git clone -b hw1-assign https://github.com/mars-research/xv6-vmx-homeworks.git
$ cd xv6-vmx-homeworks

(If you prefer SSH: git@github.com:mars-research/xv6-vmx-homeworks.git.) Commit your work to this branch as you go. The stripped functions that you write are marked with HW1 part N comments, and the files you will touch most are vmx.c, vmxasm.S and runvm.c.

Where to run it

There are two ways to run your hypervisor. Use QEMU if you can, because it is fast; use Bochs if you cannot.

QEMU with KVM (CADE machines)

QEMU needs a VMX-capable Intel CPU with nested KVM, such as the CADE machines. First put the course’s QEMU 9.0.4 on your PATH (once per shell; you can add the line to your ~/.bashrc):

$ export PATH=/home/cs5460/qemu-9.0.4/bin:$PATH

Then, in the top directory of the repo:

$ make qemu-nox

This boots xv6 on the serial console, and you get the xv6 shell. Type runvm helloguest at the $ prompt. The Makefile passes -enable-kvm -cpu host. Useful variations:

• make qemu-nox CPUS=4 gives the machine four CPUs (the default is 2);
• make qemu-nox QEMUKVM= runs without KVM. Then every CPU prints no VMX support and runvm fails cleanly. Use it to check that your vmxinit() degrades gracefully;
• to quit QEMU, press Ctrl-a then x.

If it does not work on your lab machine. QEMU can only run your hypervisor if the machine has nested virtualization turned on and lets you open /dev/kvm. If every CPU prints no VMX support even though your code is right, or QEMU complains about KVM permissions, run

$ cat /sys/module/kvm_intel/parameters/nested
$ ls -l /dev/kvm

The first should print 1 or Y, and the second should show a device that you can read and write (permissions crw-rw-rw-, or your user or group). If not, try another lab machine and tell the TAs which machine it was; or use Bochs below.

Bochs (any other machine, such as an Apple Silicon Mac)

Bochs emulates VMX in software, so it runs anywhere. It is much slower than KVM: expect the boot to take a while. The tools come from Nix.

1. Install Nix (once) from https://nixos.org/download, and enable flakes by adding this line to ~/.config/nix/nix.conf (create the file if needed):

   experimental-features = nix-command flakes
   

2. Enter the development shell. In the top directory of the repo:

   $ nix develop
   

   The first time, this downloads a cross compiler, Bochs and gdb, which takes a few minutes and needs a network connection. When it finishes you get a shell in which make, the compiler and bochs all work. Run all the commands below inside it. Leave it with exit.

3. Build and run.

   $ make bochs-nox
   

   This builds kernel, fs.img and xv6.img and starts Bochs without a window. The xv6 console is connected to your terminal, so you see the kernel’s output and can type at the xv6 shell. Quit Bochs with Ctrl-A x (Ctrl-C goes to xv6, not to Bochs). Bochs also writes a log to bochsout.txt. (make bochs opens a window instead, if your Bochs has a display; we have only tested bochs-nox.)

4. Run the test. At the xv6 shell prompt, type runvm helloguest.

Two things that can trip you up under Bochs:

• If Bochs is killed abruptly, it can leave files named fs.img.lock and xv6.img.lock behind. make bochs-nox deletes them before it starts Bochs; if you run bochs yourself and it panics with Identify: Sector Size of 0 is in error, delete both *.lock files and run again.
• Bochs is configured with two CPUs, and both print to the same console, so lines from the boot messages may be mixed together.

On a working system the test prints:

$ runvm helloguest
Hello from a VT-x guest
runvm: guest halted at rip 0x10004E

Debugging

For gdb (including its text interface), VS Code and the Bochs debugger, see the shared debugging guide.

The big picture

This section is the whole design, in the order in which a run uses it. The figure below follows one run from boot to the end, with runvm on the left calling into the kernel, and the guest on the right. The tags P1 to P6 say which part of the handout is about a function (P2 = Part 2); a function without a tag is given to you complete. After the figure, each phase gets a short explanation. Skim the section now, and come back to it when something in a part does not fit together.

Two halves, like QEMU and KVM. A user program, runvm, plays the role of QEMU: it reads the guest image, creates a VM, sets the guest’s first instruction, and emulates the guest’s devices. The kernel, in vmx.c, plays the role of KVM: it runs the guest, and handles only the VM exits it must. Everything else is returned to runvm in a struct vmexit. The kernel never opens files on behalf of the hypervisor: runvm reads the image and passes it in.

Terminology: the host is xv6 with the hypervisor (VMX root mode); the guest runs in VMX non-root mode. A VM exit is a transition from the guest to the host; a VM entry is the transition back.

The guest: helloguest

The program that runs inside the VM is called helloguest. It is small on purpose, so that you can see everything it does:

• It is not an xv6 user program. It has no ELF header, no system calls, and no operating system under it. It is a flat binary: the first byte of the file is the first instruction. The linker script helloguest.ld links it at 1 MB (VM_LOAD_ADDR, 0x100000), and the Makefile turns it into a raw binary with objcopy -O binary and puts it in the disk image fs.img, next to the other xv6 programs. runvm reads the file, and the kernel copies it to guest-physical address 0x100000.
• It starts the way a PC starts an operating system: in 32-bit protected mode with paging off. A short assembly trampoline (helloguestasm.S) switches the CPU to 64-bit mode, the same steps that xv6-64’s own entry.S takes. Part 6 walks through it.
• Then it calls guestmain() in helloguest.c, which is plain C. It loops over the string Hello from a VT-x guest\n and sends each character to the serial port with outb (a write to port COM1, 0x3F8). The guest has no serial port: every outb makes the CPU stop the guest (a VM exit), and runvm plays the part of the serial port by printing the byte. So the greeting costs 24 VM exits, one per character.
• When the string is finished, the trampoline executes hlt. That is another VM exit, and runvm takes it as “the guest is done”: it prints runvm: guest halted at rip 0x10004E (the address of the hlt) and exits.

The guest is a good first test because it uses almost nothing: a few control registers, port I/O, two MSR accesses, and one hlt. When it works, the hypervisor can run real code, and the next assignment gives it a much bigger guest.

Phase 1: boot, entering VMX operation (Part 1)

Intel’s VT-x is off until software turns it on, separately on every CPU. At boot each CPU checks for VMX, makes sure the firmware allows it, sets the control registers to values VMX accepts, and executes VMXON. From then on the CPU can run guests. On a CPU without VMX, the CPU says so and the hypervisor refuses to work, but xv6 still boots.

Phase 2: creating a VM (Parts 2 and 3)

runvm reads the guest image and calls vmcreate(), a system call. A VM is two things. The EPT (extended page table) is a page table that maps the guest’s idea of physical memory onto real pages that the kernel allocated; each page is zeroed, and the guest image is copied in at guest-physical address VM_LOAD_ADDR. The VMCS is a 4 KB structure that holds everything the CPU needs to enter and leave the guest: the guest’s state (registers, segments, control registers), the host state to restore on an exit, and control fields that say which events stop the guest.

The guest starts in the state xv6’s own boot code expects: 32-bit protected mode with paging off, which the “unrestricted guest” feature allows. Its first instructions then switch to 64-bit mode, as entry.S does on real hardware. runvm then points the guest’s RIP at the first byte of the image with vmsetreg().

Phase 3: entering and leaving the guest (Parts 4 and 5)

runvm now calls vmrun() in a loop. vmrun() makes the VMCS current, writes the host state (which depends on which CPU and process we are on), copies the guest’s RIP, RSP, RFLAGS and EFER into the VMCS, and calls vmx_enter().

vmx_enter() is assembly, because the CPU does not save or restore the general-purpose registers: it saves the host’s callee-saved registers, tells the CPU where the host stack is, loads the guest’s registers, and executes VMLAUNCH. When the guest exits, the CPU lands at a fixed address (vmx_exit) with the host stack restored. vmx_exit saves the guest’s registers and returns, so from C vmx_enter() looks like an ordinary function that returns when the guest stops.

After the exit, vmrun() reads the reason and saves the guest’s state. It then releases the VMCS (VMCLEAR) before interrupts are enabled again, so that the process may move to another CPU without leaving a stale cached VMCS behind. A consequence: every entry is a VMLAUNCH, never a VMRESUME.

Phase 4: handling an exit (Parts 5 and 6)

The kernel handles host interrupts, NMIs and accesses to the EFER register itself (vmxhandle() and vmxmsr(), [P5]) and goes around the inner loop. For every other exit it decodes what it can (for port I/O: the port, the size, the direction), advances the guest’s RIP past the instruction, and leaves the loop: vmrun() returns to runvm with the struct vmexit filled in.

runvm is the other half of the protocol. It looks at e.reason. For port I/O it emulates a transmit-only serial port: an OUT to COM1 prints the byte, and an IN from the status register reports “ready”. For an IN, runvm writes the result into the guest’s RAX with vmgetreg() and vmsetreg(), so all register changes go through one interface. A HLT means that the guest is done. Then runvm calls vmrun() again.

Here is the whole exchange for a single outb that the guest executes to print one character, from the exit to the guest resuming:

An IN is the same, with one extra step: before calling vmrun() again, runvm writes the value the device returns into the guest’s RAX with completein().

When the process exits, exit() calls vmxfreeproc(), which frees the process’s VMs.

So there are three loops, and it helps to keep them apart:

• the outer loop is in runvm: it calls vmrun() again after emulating each exit that reached user space;
• the inner loop is in the kernel’s vmrun(): it goes around for the exits the kernel handles itself, and leaves only for the others;
• the guest itself runs between vmx_enter() and vmx_exit().

What it does not do

There is one virtual CPU per VM, and no interrupt injection into the guest. String I/O (INS/OUTS) is not supported. Only the EFER MSR is emulated; any other MSR access and CPUID go back to runvm as unhandled exits. There is no VPID, so the guest’s TLB entries are flushed on every entry and exit, and the VMCLEAR on every exit trades speed for simplicity. The long-term goal is to run xv6 itself as a guest, and later assignments take it step by step toward that goal.

Files

The system call interface

int vmcreate(void *image, int len, int memsz);  // returns a VM id
int vmsetreg(int vm, int reg, uint64 val);      // VMREG_RAX..R15, RSP, RIP, RFLAGS
int vmgetreg(int vm, int reg, uint64 *val);     // read a register as of the last exit
int vmrun(int vm, struct vmexit *exit);         // run until an exit for user space

A VM id works like a file descriptor. A process owns its VMs, and exit() frees them. The kernel supports NVM (4) VMs at a time, each with up to VM_MAXMEM (64 MB) of guest memory.

struct vmexit (in vmxapi.h) is how the kernel describes an exit: the reason (EXIT_IO, EXIT_HLT, EXIT_EPT_VIOLATION, …), the exit qualification, the guest RIP, and exit-specific details. For port I/O it holds the port, the size, the direction, and, for an OUT, the value.

Main data structures

struct vm (vmx.c): one per VM, in a static table protected by a lock. Here is its definition, with the long comment shortened:

/*
 * Everything the kernel remembers about one VM.  The hardware-visible
 * state lives in the VMCS page; the fields here are the parts that we
 * keep in ordinary memory between vmrun() calls.
 */
struct vm {
  int used;                /* slot in use; protected by vms.lock */
  struct proc *owner;      /* only this process may use the VM */
  char *vmcs;              /* VMCS page (4 KB, 4 KB-aligned by kalloc) */
  uint64 *ept;             /* EPT PML4 (kernel virtual address) */
  uint64 eptp;             /* EPT pointer: PML4 phys addr | type | walk */
  uint64 memsz;            /* bytes of guest-physical memory */
  /*
   * Guest state kept here between runs, and copied to and from the
   * VMCS around each entry.  The general-purpose registers are not in
   * the VMCS at all: the processor leaves them alone across entry and
   * exit, so vmxasm.S loads and saves them by hand from/to regs.
   * RIP, RSP, RFLAGS and EFER do live in the VMCS (guest-state area).
   */
  struct guestregs regs;
  uint64 rip, rsp, rflags, efer;
  uint flushed;            /* CPUs where INVEPT has run for this EPTP */
};

The fields, one by one:

• used: whether this slot of the table is taken.
• owner: the process that may use the VM.
• vmcs: the VMCS page, a 4 KB page that the CPU uses to hold the guest and host state and the control settings.
• ept, eptp, memsz: the Extended Page Table (guest-physical to host-physical translation), the EPT pointer value for the VMCS, and the guest memory size.
• regs (struct guestregs): the guest’s general-purpose registers while the guest is not running.
• rip, rsp, rflags, efer: the other guest state, kept in memory between runs.
• flushed: a bit per CPU recording that INVEPT has run there.

VMCS: the hardware structure that holds everything about one VM. You access it only with VMREAD and VMWRITE, on the CPU where it is current (loaded with VMPTRLD). It has four groups of fields: control fields (what causes exits), guest state, host state (what the CPU loads on exit), and read-only exit information.

EPT: a four-level page table, like an ordinary x86-64 one, that translates guest-physical addresses to host-physical ones. The guest never sees host memory that is not mapped in it.

Part 1: Entering VMX operation

In this part you make every CPU enter VMX operation. Nothing runs a guest yet. When you finish, xv6 prints one line per CPU when it boots:

cpu1: VMX enabled
cpu0: VMX enabled

(With several CPUs printing at once, the lines can come out mixed together. That is normal.)

An Intel CPU has a hardware virtualization extension called VT-x, but it is off until software turns it on. Software has to do five things, in order:

1. check that the CPU supports VMX;
2. check that VMX is enabled (the firmware may have forbidden it);
3. put the control registers in a state that VMX accepts;
4. prepare a page of memory for VMXON to use as bookkeeping;
5. execute the VMXON instruction.

All of this happens in one function, vmxinit(), which mpmain() already calls on every CPU. Each CPU has to enter VMX operation separately, because VMX is a per-CPU feature.

Where to look things up. Each step below lists the sections of the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (System Programming Guide, order number 325384-092, June 2026), that describe the mechanism it uses. We cite that edition throughout, and write “SDM” for it. Read the sections: the manual is the real reference, and this handout only walks you through it. Section numbers change between editions, so if you use another one, search for the titles.

Open vmx.c. You will find vmxinit() with an empty body and a few things above it that we give you:

• vmx_ok, a field of struct cpu (in proc.h) that the rest of the hypervisor checks, per CPU, before it does anything;
• vmcs_revision, a global that we will need in the next part;
• the wrappers cpuid_insn(), vmxon(), and, in x86.h, rdmsr(), wrmsr(), rcr0(), lcr0(), rcr4() and lcr4();
• the constants (MSR_IA32_FEATURE_CONTROL, CR4_VMXE, …) in vmx.h.

You will build the function in five steps. After each block of code, we explain what it does. Paste the blocks in order into the body of vmxinit().

Step 1: Check that the CPU supports VMX

SDM: 26.6 “Discovering Support for VMX”; Vol. 2A, the CPUID instruction.

  uint a, b, c, d;
  uint64 fc, cr0, cr4;
  int id = cpuid();
  char *region;

  /* CPUID.1:ECX.VMX[bit 5] says whether the CPU has VT-x at all. */
  cpuid_insn(1, &a, &b, &c, &d);
  if(!(c & CPUID1_ECX_VMX)){
    cprintf("cpu%d: no VMX support\n", id);
    return;
  }

xv6 requires declarations at the top of a function. a, b, c and d hold the four registers that the CPUID instruction returns. fc, cr0 and cr4 will hold values we read and modify in later steps. id is the number of the CPU we are running on, and region will point to this CPU’s VMXON page, which we allocate in Step 4.

CPUID leaf 1 reports the CPU’s features. Bit 5 of ECX is set if the CPU supports VMX. If it is not, we print a message and return without setting cpus[id].vmx_ok. The rest of the hypervisor sees the flag clear and refuses to work, so the system is still usable.

This check must come first. The next step reads a model-specific register that exists only if VMX does. On a CPU without VMX, rdmsr of a missing register raises a fault, and in the kernel a fault is a panic.

Step 2: Check that VMX is enabled

SDM: 26.7 “Enabling and Entering VMX Operation”, which describes the bits of the IA32_FEATURE_CONTROL MSR.

  /*
   * IA32_FEATURE_CONTROL gates VMXON.  Once its lock bit is set it
   * cannot be changed until reset, and VMXON faults unless the
   * "enable VMX outside SMX" bit is set.  The firmware normally
   * enables VMX and locks this MSR; if it left the MSR unlocked we
   * can set the bits ourselves.  This MSR exists only when CPUID
   * reports VMX, so the read is safe here.
   */
  fc = rdmsr(MSR_IA32_FEATURE_CONTROL);
  if(!(fc & FEATURE_CONTROL_LOCKED)){
    fc |= FEATURE_CONTROL_LOCKED | FEATURE_CONTROL_VMXON_OUTSIDE_SMX;
    wrmsr(MSR_IA32_FEATURE_CONTROL, fc);
  }
  if(!(fc & FEATURE_CONTROL_VMXON_OUTSIDE_SMX)){
    cprintf("cpu%d: VMX disabled by firmware\n", id);
    return;
  }

A CPU can support VMX and still have it switched off. The MSR IA32_FEATURE_CONTROL is the switch, and the firmware (BIOS) normally sets it before the OS starts. Two of its bits matter to us:

• bit 0, “lock”. If it is clear, VMXON faults. Once it is set, the MSR cannot be changed until the machine is reset, so the firmware sets it to make its choice final.
• bit 2, “enable VMX outside SMX”. VMXON faults unless this is set. (SMX is a separate Intel security feature that we don’t use; bit 1 is its counterpart.)

There are three cases:

The code handles this with two ifs. The first fixes the unlocked case by setting both bits, which also updates fc, so the second if sees the new value. The second catches the locked-off case.

Step 3: Make CR0 and CR4 legal for VMX

SDM: 26.8 “Restrictions on VMX Operation”; Appendix A.7 “VMX-Fixed Bits in CR0” and A.8 “VMX-Fixed Bits in CR4”.

  /*
   * In VMX operation CR0 and CR4 must obey the fixed-bit MSRs
   * (for example CR0.NE = 1 and CR4.VMXE = 1).  FIXED0 lists the bits
   * that must be 1 (a bit set in FIXED0 must be 1), and FIXED1 lists
   * the bits that may be 1 (a bit clear in FIXED1 must be 0).  So OR
   * in FIXED0, then AND with FIXED1.  VMXON fails otherwise
   * (SDM 26.8).  CR4.VMXE must be set before VMXON is executed.
   */
  cr0 = rcr0();
  cr0 |= rdmsr(MSR_IA32_VMX_CR0_FIXED0);
  cr0 &= rdmsr(MSR_IA32_VMX_CR0_FIXED1);
  lcr0(cr0);
  cr4 = rcr4() | CR4_VMXE;
  cr4 |= rdmsr(MSR_IA32_VMX_CR4_FIXED0);
  cr4 &= rdmsr(MSR_IA32_VMX_CR4_FIXED1);
  lcr4(cr4);

In VMX operation, some bits of CR0 and CR4 are not yours to choose. Two MSRs per register tell you which:

• FIXED0: a bit that is 1 here must be 1 in the control register.
• FIXED1: a bit that is 0 here must be 0 in the control register.

So the legal value is “OR in FIXED0, then AND with FIXED1”. The code does exactly that for both registers and writes the result back.

Two examples: CR0.NE (numeric error) must be 1 in VMX operation, and CR4.VMXE must be 1 to execute VMXON at all. We set CR4.VMXE ourselves on the first line of the CR4 group, so we don’t depend on it being in FIXED0.

You will see this pattern again. The VMCS control fields in part 3 use the same idea, with “allowed-0” and “allowed-1” halves in one MSR. If you understand this step, that one is easy.

Step 4: Prepare the VMXON region

SDM: 27.11.5 “VMXON Region”; Appendix A.1 “Basic VMX Information” (the IA32_VMX_BASIC MSR).

  /*
   * The VMXON region must start with the VMCS revision identifier
   * (bits 30:0 of IA32_VMX_BASIC; bit 31 of the region must be 0).
   * VMXON also requires the region to be 4 KB-aligned (SDM 27.11.5),
   * which a kalloc() page is.  Zero it first so the rest is
   * deterministic.  Each CPU keeps its region while VMX is on.
   */
  vmcs_revision = rdmsr(MSR_IA32_VMX_BASIC) & 0x7FFFFFFF;
  if((region = kalloc()) == 0){
    cprintf("cpu%d: no memory for the VMXON region\n", id);
    return;
  }
  memset(region, 0, PGSIZE);
  *(uint*)region = vmcs_revision;

VMXON needs a 4 KB-aligned page of memory that the CPU uses for its own bookkeeping. The only rule on its contents is that the first four bytes hold the VMCS revision identifier, a 31-bit number that identifies the CPU’s VMCS format. It lives in bits 30:0 of the MSR IA32_VMX_BASIC (bit 31 of the region must be 0, which the mask takes care of). The manual says software “need not initialize the VMXON region in any other way” and should use a separate region for each CPU.

We get the region from kalloc(), which returns a page-aligned 4 KB page, as VMXON requires. kalloc() does not zero the page. The CPU does not need the rest of the region to be zero, but we zero it anyway, so that nothing stale from a previous use of the page is left in it, and then write the revision into its first four bytes.

We also save the revision in the global vmcs_revision. You will use it again in part 2, because every VMCS page starts with the same number.

Is it safe to allocate here, on every CPU? It is. startothers() brings the other CPUs up one at a time and waits for each to finish mpmain() (which calls vmxinit()) before it continues, so no two CPUs allocate at once, even though the allocator’s lock is not enabled yet.

Step 5: Execute VMXON

SDM: 26.7; Chapter 33, “VMXON–Enter VMX Operation”.

  *(uint*)region = vmcs_revision;
  /* VMXON takes the region's physical address. */
  if(vmxon(V2P(region)) < 0){
    cprintf("cpu%d: vmxon failed\n", id);
    kfree(region);
    return;
  }
  cpus[id].vmx_ok = 1;
  cprintf("cpu%d: VMX enabled\n", id);

VMXON takes the region’s physical address, which is why the code calls V2P(): the kernel works with virtual addresses, but the hardware uses physical ones. The wrapper vmxon() returns a negative number if the instruction reports failure. On failure we free the page, since nothing will use it.

If it succeeds, we set cpus[id].vmx_ok. Every later function in the hypervisor checks this flag first. Only then do we print the success message.

Checkpoint

Build and boot (see the “Where to run it” section of this handout for QEMU and Bochs). Every CPU should print:

cpuN: VMX enabled

Two more things to try:

1. Boot with make qemu-nox QEMUKVM= (no VMX). Each CPU should print no VMX support, and the system should still reach the shell. If it panics instead, check Step 1.
2. Optional experiment. Move Step 2 above Step 1 and boot with QEMUKVM=. Explain what you see.

If something fails:

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. In Step 1, why does the function return when VMX is missing instead of calling panic()?
2. In Step 2, on most machines the firmware has already enabled and locked VMX. Why does the code still handle the unlocked case instead of just failing?
3. In Step 3, what would happen if we skipped lcr4(cr4) and went straight to VMXON?
4. In Step 4, why does each CPU need its own VMXON region? What could go wrong if all CPUs shared one?

Part 2: Creating a VM, its memory, and the EPT

In this part you build the part of vmcreate() that gives a VM its memory. When you finish, runvm can create a VM, and the kernel has copied the guest program into guest memory. The guest does not run yet: the VMCS is still empty, and filling it in is part 3. So at the end of this part you check your work by looking at the kernel’s own state (a temporary cprintf), not by watching the guest.

Part 1 gave every CPU VMX operation. Now we need three things for one VM:

1. a slot in the kernel’s table of VMs;
2. a VMCS page, the 4 KB structure that the CPU will later use to hold the guest’s and the host’s state (we only allocate it here);
3. guest-physical memory: a block of RAM that the guest will believe is its own physical memory, and the EPT, the page table that tells the CPU which real page backs each guest-physical page.

What is guest-physical memory?

A normal OS sees physical addresses that go straight to RAM. A guest OS believes the same, but it must not be able to touch the host’s RAM. So the hypervisor puts one more translation in the middle:

The guest controls the first step with its own page tables, exactly as on real hardware. Only the hypervisor controls the second step. The guest has no way to name a host page that the EPT does not map. When the guest touches a guest-physical address that the EPT does not map (or maps without the needed permission), the CPU takes an EPT violation VM exit and the host decides what to do.

The EPT has the same shape as the x86-64 page tables you know from xv6: four levels, 512 entries of 8 bytes in each 4 KB table, 9 bits of address per level. The differences are in the entry format, which you will see in Section A.

Where to look things up. As in part 1, each step lists the sections of Intel SDM Volume 3 (order number 325384-092, June 2026) that describe what it uses.

What is given and what you write

Open vmx.c. These pieces are given:

• struct vm, the static table vms (with NVM = 4 slots and a lock), and the global vmcs_revision from part 1;
• getvm(), which turns a VM id into a struct vm * and checks that the caller owns it. Later parts and the system calls use it; this part does not call it;
• vmfree() and eptfree(), which release a VM and, recursively, the EPT tables and every guest page they map;
• the wrappers vmclear() and vmptrld(), the EPT constants in vmx.h (EPT_RWX, EPT_MEMTYPE_WB, EPT_ADDR(), EPTP_*, EPT_CAP_*), and the system call glue in sysvmx.c and runvm.c.

You write two functions:

(The build uses -Werror, but your Makefile turns off the unused-variable and unused-function warnings, so the stubs compile even though they leave variables and helper functions unused. Everything else is still an error.)

Section B also contains the last block of vmcreate(), which calls vmcsinit(). We give you that block in full, since it is short, but its meaning is part 3. In part 3 you write vmcsinit() itself.

Section A: eptwalk()

eptwalk(pml4, gpa, alloc) returns a pointer to the leaf EPT entry for guest-physical address gpa: the 8-byte slot in a page table that says which host page backs this 4 KB of guest memory. If a table on the way is missing and alloc is non-zero, it creates the table. If a table is missing and alloc is zero, or memory runs out, it returns 0. vmcreate() calls it twice: with alloc set to build the mapping, and with alloc clear to find the host page again when it copies the image.

The entry format

Open vmx.h and read the EPT section. An EPT entry is a 64-bit number:

 63          52 51                     12 11  7  6  5   3  2  1  0
+--------------+-------------------------+-----+---+------+--+--+--+
|   (ignored)  |  physical address >> 12 |     |   |  mt  | X| W| R|
+--------------+-------------------------+-----+---+------+--+--+--+

• Bits 2:0 are R, W, X: read, write and instruction-fetch permission for the region the entry controls. This is different from an ordinary page table, where bit 0 means “present”. An EPT entry is present if any of bits 2:0 is 1. We call “R | W | X” EPT_RWX.
• Bits 51:12 hold the physical address of the next-level table (for an upper-level entry) or of the 4 KB host page (for a leaf entry). EPT_ADDR(e) extracts them.
• Bits 5:3 of a leaf entry hold the EPT memory type (6 = write-back). In an entry that points to a lower table these bits are reserved and must be zero. We will use this in Section B.

The effective permission of an access is the AND over all the levels it passed through. So if an upper-level entry denies an access, no leaf entry can allow it. We use that: upper-level entries always get EPT_RWX, and only the leaf says anything restrictive.

Step A1: Walk the three upper levels

SDM: 31.3.2 “EPT Translation Mechanism” (the paragraphs for the EPT PML4E, PDPTE and PDE, and Tables 31-2 to 31-6).

  uint64 *table = pml4, *e;
  char *next;
  int level;

  /*
   * Walk levels 3 (PML4), 2 (PDPT) and 1 (PD); level 0, the page
   * table, is indexed after the loop.  Index = 9 bits starting at
   * bit 12 + 9*level.
   */
  for(level = 3; level > 0; level--){
    e = &table[(gpa >> (12 + 9*level)) & 0x1FF];
    if(*e & EPT_RWX){
      /* Present: descend.  The entry holds a physical address. */
      table = (uint64*)P2V(EPT_ADDR(*e));
    } else {
      if(!alloc || (next = kalloc()) == 0)
        return 0;
      /* kalloc() doesn't zero; a non-zero junk entry would look present. */
      memset(next, 0, PGSIZE);
      /*
       * Non-leaf entries grant R/W/X so that only the leaf entry
       * restricts permissions (the effective permission is the AND
       * over all levels).
       */
      *e = V2P(next) | EPT_RWX;
      table = (uint64*)next;
    }
  }

level counts from 3 (the PML4) down to 1 (the page directory). The page table, level 0, is indexed after the loop (Step A2). In each table, 9 bits of the guest-physical address select one of the 512 entries. The shift is 12 + 9*level, so the three indexes come from bits 47:39, 38:30 and 29:21. The manual gives the same ranges (bits 47:39 select the PML4E, and so on). Bits 11:0 are the offset inside the 4 KB page.

For one entry there are two cases.

• The entry is present. Its bits 51:12 are the physical address of the next table. The kernel can only dereference virtual addresses, so we convert with P2V() and continue. (EPT_ADDR() masks away the permission bits, which would otherwise corrupt the address.)
• The entry is not present. If the caller did not ask us to create tables, or kalloc() fails, return 0. Otherwise take a fresh page, zero it, store its physical address in the entry together with EPT_RWX, and continue into it.

The memset is not optional. kalloc() does not clear pages. xv6’s kfree() fills a freed page with the byte 0x01 to catch dangling pointers, so a recycled page is full of non-zero entries. Every 0x01... entry has bit 0 set, so it would look present and point at a bogus address. The same rule holds for xv6’s own page tables: a new table must be all zero before use.

Note the order in the not-present case: we fill the new table before we link it into *e. And if kalloc() fails halfway through a multi-level build, the tables already linked are still reachable from the PML4, so eptfree() will find and release them. Nothing leaks.

Step A2: Return the leaf entry (Your turn)

SDM: 31.3.2, Table 31-7 “Format of an EPT Page-Table Entry that Maps a 4-KByte Page”.

After the loop, table points to the EPT page table (the lowest level), which holds 512 leaf entries, each mapping one 4 KB page.

Goal. Return a pointer to the entry in this table that corresponds to gpa. Do not read or modify it: the caller decides what to store.

Hints.

• Which bits of gpa select the entry in the lowest table? Table 31-7’s text says an EPT PTE is identified using bits 47:12 of the address, and the bits that select within one table are the lowest nine of those.
• This is the same index computation as in the loop, with level = 0.
• Return the address of the entry, as the function’s return type uint64 * and the comment above it say.

Section B: vmcreate()

vmcreate(image, len, memsz) is called by the system call sys_vmcreate() (in sysvmx.c). By then image is a kernel copy of the guest binary (the system call code checked the user pointer). It returns a VM id, which works like a file descriptor, or -1. Paste the blocks below in order, after the declarations.

Step B1: Check the arguments

SDM: none; these are the limits of this implementation. See VM_MAXMEM and VM_LOAD_ADDR in vmxapi.h.

  /* The image must fit in guest memory above its load address. */
  if(memsz <= 0 || memsz > VM_MAXMEM || len < 0 || VM_LOAD_ADDR + len > memsz)
    return -1;

memsz must be positive and at most VM_MAXMEM (64 MB). Every page of guest memory comes out of the kernel’s own memory, so a user must not be able to ask for all of it. The image is loaded at guest-physical VM_LOAD_ADDR (1 MB), so it fits only if VM_LOAD_ADDR + len <= memsz. A negative len is rejected too. Check everything before allocating anything: early checks can just return -1, while later failures need cleanup.

Step B2: Check that this CPU is in VMX operation

SDM: 26.7 “Enabling and Entering VMX Operation”.

  /*
   * The VMX capability MSRs don't exist without VMX; reading them faults.
   * pushcli() keeps us on one CPU while we look at its flag.
   */
  pushcli();
  r = mycpu()->vmx_ok;
  popcli();
  if(!r)
    return -1;

This is the flag you set in part 1. We must not go on without it, because the next step reads a VMX capability MSR, and on a machine without VMX that read faults and panics the kernel. (The same warning appears in Part 1, Step 1.)

mycpu() is only valid while the process cannot move to another CPU, so we wrap it in pushcli() and popcli() (they disable and re-enable interrupts). We copy the flag into r and test it after popcli(). There is a related use of pushcli() at the end of vmcreate(); there it protects something more subtle.

Step B3: Check the EPT capabilities

SDM: Appendix A.10 “VPID and EPT Capabilities” (the IA32_VMX_EPT_VPID_CAP MSR); 27.6.11 “Extended-Page-Table Pointer (EPTP)”.

  /*
   * We need: 4-level EPT walks, write-back EPT memory type, INVEPT
   * and its single-context type (SDM Appendix A.10).
   */
  need = EPT_CAP_WALK_4 | EPT_CAP_MEMTYPE_WB | EPT_CAP_INVEPT | EPT_CAP_INVEPT_SINGLE;
  if((rdmsr(MSR_IA32_VMX_EPT_VPID_CAP) & need) != need){
    cprintf("vmx: CPU lacks required EPT features\n");
    return -1;
  }

Not every VMX CPU supports every EPT option. IA32_VMX_EPT_VPID_CAP is a bit mask of what is available. We need four bits, which A.10 describes:

• bit 6: a page-walk length of 4 (our EPT has four levels);
• bit 14: write-back as the EPT paging-structure memory type;
• bit 20: the INVEPT instruction exists;
• bit 25: INVEPT supports the single-context type (part 4 uses this to flush the cached translations of one EPT).

The test (msr & need) != need means “at least one of the needed bits is missing”. (The odd spacing in need =EPT_CAP... is in the source; keep it, it compiles.)

This MSR exists only on processors that support the “activate secondary controls” and “enable EPT” controls, which is another reason the vmx_ok check comes first.

Step B4: Claim a slot in the VM table

SDM: none; this is bookkeeping in the kernel.

  /* Claim a free VM slot under the lock. */
  acquire(&vms.lock);
  for(id = 0; id < NVM; id++)
    if(!vms.vm[id].used)
      break;
  if(id == NVM){
    release(&vms.lock);
    return -1;
  }
  vm = &vms.vm[id];
  /* Zero everything: null pointers (for vmfree), flushed = 0, regs = 0. */
  memset(vm, 0, sizeof(*vm));
  vm->used = 1;
  vm->owner = myproc();
  release(&vms.lock);

  vm->memsz = PGROUNDUP(memsz);
  vm->rflags = 0x2;  /* bit 1 is reserved and must be 1 */

The table vms.vm[] is shared by every process, so a free slot is found and marked used while holding the lock. Otherwise two processes could pick the same slot. If there is no free slot, release the lock and fail.

Then the slot is wiped with memset, for two reasons. All the pointers (vmcs, ept) become null, which is what vmfree() relies on to know what has been allocated so far. And flushed, regs and the saved registers start at zero. After that the slot is ours: used is set and owner is the calling process, which is how getvm() later checks ownership.

We release the lock before doing the slow work (allocation, copying a whole image) and keep the slot reserved by used instead.

memsz is rounded up to a whole number of pages. rflags is the guest’s initial RFLAGS: bit 1 of RFLAGS always reads as 1 on x86, and VM entry will check for it (part 3).

From this point on, every failure must goto bad, not return -1. The slot is already marked used, and bad: (Step B9) is where vmfree() releases whatever has been allocated so far and marks the slot free. A bare return -1 here would leak the slot; four such failures and vmcreate() would never succeed again until reboot.

Step B5: Allocate the VMCS page

SDM: 27.2 “Format of the VMCS Region”.

  /*
   * The VMCS page.  Zero it: VMCS contents are undefined to software,
   * but starting from zeros keeps the revision-id word and abort
   * indicator clean (the revision id is set below).
   */
  if((vm->vmcs = kalloc()) == 0)
    goto bad;
  memset(vm->vmcs, 0, PGSIZE);

The VMCS is a region of memory of at most 4 KB, whose location you hand to VMPTRLD. VMPTRLD fails with “invalid physical address” if the address is not 4 KB aligned (Chapter 33), and kalloc() guarantees the alignment. We only allocate and clear it here. The first four bytes will hold the revision identifier (Step B9), and the next four are the VMX-abort indicator, which Table 27-1 says the processor writes if a VMX abort occurs. The rest is “VMCS data” in an implementation-specific format that you never touch directly, only through VMREAD and VMWRITE in part 3. We zero the page so that all the words we do control start out clean.

Step B6: Allocate the EPT root and build the EPTP

SDM: 27.6.11 “Extended-Page-Table Pointer (EPTP)”, Table 27-9; 31.3.7.1 “Memory Type Used for Accessing EPT Paging Structures”.

  /* The EPT PML4: must be zeroed, since zero entries mean "not present". */
  if((vm->ept = (uint64*)kalloc()) == 0)
    goto bad;
  memset(vm->ept, 0, PGSIZE);
  /*
   * EPT pointer (SDM 27.6.11): physical address of the PML4 in the
   * high bits, EPT paging-structure memory type in bits 2:0 (6 =
   * write-back), and page-walk length minus one in bits 5:3 (3 = a
   * 4-level walk).
   */
  vm->eptp = V2P(vm->ept) | EPTP_WALK_4 | EPTP_MEMTYPE_WB;

vm->ept is the EPT’s PML4 (the root table), allocated and zeroed like every table: a zero entry means “not present”, and eptwalk() relies on that.

The VMCS does not hold a pointer to the PML4 directly. It holds the EPT pointer (EPTP), a 64-bit value with the root’s physical address in bits 51:12 and configuration in the low bits (Table 27-9):

• bits 2:0: memory type used by the CPU when it reads the EPT tables (6 = write-back, 0 = uncacheable). EPTP_MEMTYPE_WB is 6.
• bits 5:3: the page-walk length minus one. A value of 3 means four levels. EPTP_WALK_4 is 3 << 3.
• bit 6 (accessed and dirty flags) stays 0. We do not use it.

We build the value now and store it in vm->eptp. In part 3, vmcsinit() writes it to the VMCS. Write-back is the sensible choice: the EPT tables are ordinary RAM, and uncacheable would make every table walk slow. It is also why Step B3 checked bit 14.

Step B7: Allocate and map the guest’s memory

SDM: 31.3.2, Table 31-7 (bits 2:0, 5:3 and 51:12 of a leaf entry); 31.3.7.2 “Memory Type Used for Translated Guest-Physical Addresses”.

  /*
   * Guest memory: zeroed 4 KB pages, readable, writable, executable,
   * write-back cacheable.  Guest-physical address gpa maps to the
   * host page "mem".  xv6's kfree() fills freed pages with junk, so
   * we zero here; otherwise the guest would see host leftovers
   * (and a security leak).  The leaf memory type (bits 5:3) is
   * valid only in leaf entries.
   */
  for(gpa = 0; gpa < vm->memsz; gpa += PGSIZE){
    if((pte = eptwalk(vm->ept, gpa, 1)) == 0 || (mem = kalloc()) == 0)
      goto bad;
    memset(mem, 0, PGSIZE);
    *pte = V2P(mem) | EPT_RWX | EPT_MEMTYPE_WB;
  }

This is the heart of the part. For every 4 KB page of guest-physical memory, from address 0 up to memsz:

1. eptwalk(..., 1) finds the leaf entry, creating any missing tables.
2. kalloc() provides a host page that will hold this guest page.
3. memset clears it.
4. The leaf entry is set to the host page’s physical address, plus permissions and a memory type.

Why zero the page? kalloc() does not clear memory, and kfree() fills pages with junk. A guest page we did not zero would contain whatever the kernel or another process had there: kernel data, someone’s file contents. The guest could read it. That is a security hole. It would also make the guest’s behavior depend on what ran earlier, which is the kind of bug that vanishes when you add a cprintf. Our own guest also relies on it: the image is a flat binary with no loader to clear its uninitialized data, so the zeroed memory beyond the image is what makes that data zero.

Why R, W and X? The guest will run code from this memory (X), write its stack and data (W), and read both (R). We do not need any restrictions in this homework. The leaf is the right place to restrict a page later (for example, to make guest memory read-only to trap writes), because the upper levels already allow everything. Be careful if you experiment with other permissions: an entry with R clear and W set is invalid and causes an EPT misconfiguration (31.3.3.1), a different and more confusing kind of exit than an EPT violation. The same holds for R clear and X set, unless the CPU supports execute-only translations (Appendix A.10, bit 0).

Why write-back? Bits 5:3 of the leaf entry give the memory type for the guest’s accesses to this page (31.3.7.2): 0 is uncacheable, 6 is write-back, and so on. The guest memory is ordinary RAM, so we want it cached. Uncacheable memory would work but run very slowly. EPT_MEMTYPE_WB is 6 << 3, already shifted into bits 5:3. Because these bits are only meaningful in a leaf entry, eptwalk() leaves them zero in upper-level entries, where they must be zero.

Why V2P(mem)? The kernel works with virtual addresses, but the EPT stores physical addresses, because the CPU walks it with real memory accesses. The same conversion is used for the VMXON region in part 1.

The loop can fail. The kernel manages physical memory below PHYSTOP (0xE000000, 224 MB), and a 64 MB guest needs 16384 data pages plus about 35 table pages. If any kalloc() fails, we goto bad, and vmfree() frees what exists. Note that the leaf entry is written only after both allocations succeeded and the page was zeroed, so eptfree() never sees a half-set entry.

Memory used per VM. A 4 MB guest needs 1024 data pages, 2 page tables, 1 page directory, 1 PDPT and 1 PML4, plus the VMCS page.

Step B8: Copy the image into guest memory

SDM: 31.3.2 (the host-physical address of a leaf is bits 51:12 of the entry, combined with bits 11:0 of the guest-physical address).

  /*
   * Copy the image in, one guest page at a time.  The guest-physical
   * range is contiguous but the host pages backing it are not, so
   * translate each page through the EPT in software.
   */
  for(off = 0; off < len; off += n){
    gpa = VM_LOAD_ADDR + off;
    /* Bytes left in this guest page, capped by the bytes left to copy. */
    n = PGSIZE - gpa % PGSIZE;
    if(n > len - off)
      n = len - off;
    /* Already populated above (the size check guarantees it). */
    pte = eptwalk(vm->ept, gpa, 0);
    memmove((char*)P2V(EPT_ADDR(*pte)) + gpa % PGSIZE, image + off, n);
  }

The guest image is a flat binary that must appear at guest-physical address VM_LOAD_ADDR. We cannot just memmove it to some kernel address: the guest-physical range is contiguous, but the host pages that back it came from kalloc() one at a time and are not contiguous (and not in order). So we do in software what the CPU does in hardware: translate each guest page through the EPT, and copy at most one page at a time.

In each iteration:

• gpa is the guest-physical address of the next byte to write.
• n is the number of bytes we can copy before crossing a page boundary (PGSIZE - gpa % PGSIZE), capped by the bytes that remain in the image. The first chunk can be shorter than a page only if the load address is not page aligned. It is aligned here, but the code does not rely on it.
• eptwalk(..., 0) finds the leaf entry. alloc is 0 because the page must already be mapped. The size check in Step B1 guarantees that every page of the image lies inside memsz, and Step B7 mapped all of it. If eptwalk() ever returned 0 here, *pte would crash. The check is what makes it safe.
• EPT_ADDR(*pte) gives the host-physical address of the page, P2V() makes it usable by the kernel, and gpa % PGSIZE is the offset inside that page.

Step B9: Prepare the VMCS (explained in part 3)

SDM: 27.2 “Format of the VMCS Region”; Chapter 33: VMCLEAR, VMPTRLD; 27.11.1 and 27.11.3 for how software uses and initializes a VMCS.

  /*
   * Prepare the VMCS.  First write the revision identifier at the
   * start of the VMCS page (SDM 27.2), without which VMPTRLD fails.
   * VMCLEAR initializes the VMCS and puts it in the "clear" launch
   * state, which is what VMLAUNCH requires.  VMPTRLD makes it current
   * so that vmcsinit()'s VMWRITEs go to it.  pushcli() keeps us on
   * this CPU, because "current" is per-CPU state.
   */
  *(uint*)vm->vmcs = vmcs_revision;
  pushcli();
  if(!mycpu()->vmx_ok || vmclear(V2P(vm->vmcs)) < 0 || vmptrld(V2P(vm->vmcs)) < 0){
    popcli();
    goto bad;
  }
  r = vmcsinit(vm);
  /*
   * Always VMCLEAR when done, even on error, so no CPU holds the
   * VMCS cached and it can be freed or loaded elsewhere.
   */
  vmclear(V2P(vm->vmcs));
  popcli();
  if(r < 0)
    goto bad;
  return id;

bad:
  vmfree(vm);
  return -1;

This is the last block, and it ends the function. You do not need to understand all of it now, and part 3 returns to it. In short:

1. Write the VMCS revision identifier (saved by vmxinit() in part 1) into the first four bytes of the VMCS page. VMPTRLD fails if it is missing or wrong (27.2).
2. With interrupts off (pushcli()), VMCLEAR the VMCS (this puts it in the “clear” state, which VMLAUNCH requires), then VMPTRLD it to make it the current VMCS of this CPU. “Current” is per-CPU state, so the process must not move to another CPU in between, which is what pushcli() ensures.
3. vmcsinit() fills the VMCS with VMWRITEs. This is part 3.
4. VMCLEAR again, always, even if vmcsinit() failed, so that no CPU still holds the VMCS and it is safe to free or load elsewhere.
5. On success return the id. On any failure, fall into bad:.

bad: is the single cleanup path. vmfree() (given) frees the EPT tree and all guest pages through eptfree(), frees the VMCS page, and only then marks the slot reusable, so a concurrent vmcreate() cannot claim a slot whose resources are still being released. It is safe on a partly built VM because Step B4 zeroed the struct: a pointer that was never assigned is null and is skipped.

Checkpoint

You cannot see the guest run yet, so you look at the kernel instead.

1. A temporary print. Add this line between Step B8 and Step B9 (remove it afterwards):

  cprintf("vmcreate: vm %d, memsz %x, eptp %x\n", id, vm->memsz, vm->eptp);

Boot and run runvm helloguest. You should see one line such as vmcreate: vm 0, memsz 400000, eptp .... For the default 4 MB guest, memsz is 400000 and the low three hex digits of eptp are 01e: the PML4 page is 4 KB aligned, so the low 12 bits come only from EPTP_WALK_4 | EPTP_MEMTYPE_WB, which is 0x18 | 0x6 = 0x1E. The upper digits are the physical address of the PML4 and vary.

(cprintf supports only %d, %x, %p and %s, and %x reads 64 bits, so keep 64-bit values as arguments to %x, or cast 32-bit ones to uint64.)

2. Is the image really in guest memory? Add, after the print:

  pte = eptwalk(vm->ept, VM_LOAD_ADDR, 0);
  cprintf("image word %x, guest word %x\n", (uint64)*(uint*)image,
          (uint64)*(uint*)P2V(EPT_ADDR(*pte)));

The two numbers must be equal. If they are not, check Step B8.

3. What runvm says. When vmcreate() succeeds, runvm gets a VM id and goes on to vmsetreg() and then vmrun(). With vmcsinit() not yet written (we expect your part 3 stub just returns without touching the VMCS) the CPU cannot start the guest, and runvm prints a message about vmrun, likely runvm: VMLAUNCH failed, VM-instruction error 7 (Table 33-1: “VM entry with invalid control field(s)”). That is the success signal for this part: you got past vmcreate(). The exact number may differ with the stub, but it should not be runvm: vmcreate failed.

If instead you see runvm: vmcreate failed, vmcreate() returned -1. The cprintf of check 1 tells you whether you got as far as Step B8.

4. Limits and cleanup.

• runvm helloguest 0 and runvm helloguest 100 (larger than 64 MB) should print runvm: vmcreate failed (Step B1).
• runvm helloguest 1 should also fail: the image is loaded at 1 MB, so a 1 MB memory has no room for it.
• Run runvm helloguest 64 five times in a row. Each run creates a VM, and exit() destroys it. If vmfree() and eptfree() work, every run behaves like the first. If pages were leaked, a later run would fail with vmcreate failed, since 5 x 64 MB is more than the 224 MB below PHYSTOP.

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. An ordinary x86-64 page table translates virtual addresses to physical addresses. What does the EPT translate, and who controls it? What would go wrong if the hypervisor gave the guest direct access to the host’s physical memory instead?
2. In Step B7 we zero every page of guest memory, and in eptwalk() we zero every new table. These are two different reasons to zero. State each, and say what the guest or the CPU would see if we skipped it.
3. In Step B8, why does vmcreate() copy the image page by page through eptwalk(), instead of one memmove of len bytes to the guest’s first page?
4. An EPT entry is “present” if any of R, W or X is set, not if one particular bit is set. A common bug is to test only bit 0. Name one kind of entry that this bug would treat as missing but that the CPU treats as valid (hint: see Appendix A.10 for execute-only support), and say why our code never creates such an entry.

Part 3: The VMCS, describing the guest and the host

In this part you fill in the VMCS, the structure that tells the CPU what a virtual machine looks like and when to stop running it. Nothing runs a guest yet (that is parts 4 and 5), but this is the part where most of the hardware’s rules are learned. When you finish, vmcreate() succeeds: runvm gets past its first system call without printing runvm: vmcreate failed, and a few values that you read back from the VMCS are the ones you wrote.

What a VMCS is. The VMCS (virtual-machine control structure) is a 4 KB region of memory that holds everything about one virtual CPU. It has four groups of fields:

• guest state: what the CPU loads when it enters the guest (CR0, CR3, segment registers, RIP, …);
• host state: what the CPU loads when the guest exits back to us;
• control fields: which events cause exits, and what to save and load at each transition;
• exit information: read-only fields the CPU fills in at an exit (part 5).

Software never reads or writes the memory directly, because its layout is different on different CPUs. Instead, it uses two instructions, VMREAD and VMWRITE, each of which takes a field encoding (a number such as GUEST_CR0 = 0x6800, listed in vmx.h). They work on the current VMCS of the CPU:

• VMPTRLD pa makes the VMCS at physical address pa the current one.
• VMCLEAR pa writes the CPU’s cached copy back to memory, marks the VMCS clear (so the next entry must use VMLAUNCH), and, if it was current, leaves the CPU with no current VMCS.

vmcreate() already does the VMCLEAR and VMPTRLD for you, writes the revision identifier that you saved in part 1 into the first four bytes of the page, and calls vmcsinit() with the VMCS current. When vmcsinit() returns, vmcreate() clears the VMCS again. Your job is the body of vmcsinit(), plus the helper adjustctl() that it calls.

Where to look things up. As before, the manual is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026), written “SDM” below. Chapter 27 describes every VMCS field, chapter 29 lists the checks the CPU makes at VM entry, and Appendix A explains the capability MSRs.

What we give you and what you write

Given (in vmx.c, vmx.h):

• the wrappers vmclear(), vmptrld(), vmread() and vmwrite(). vmwrite() panics if the instruction fails, after printing the VM-instruction error number (Table 33-1 in SDM 33.4), so a typo in a field name is reported at once;
• guestseg(), shown below;
• all the field encodings and control-bit constants (GUEST_CR0, PIN_EXTINT_EXITING, AR_CODE32, …);
• struct vm, which holds vm->eptp, the EPT pointer that vmcreate() built;
• vmxhoststate() and vmrun(), which part 4 and 5 are about.

You write:

In vmcsinit() all the blocks are pasted in order into the function body. This part has no “Your turn” exercise. The code is there to be read: for each block, find the matching field in the SDM and check that you understand why the value is what it is.

guestseg() is given. It writes the four VMCS fields that describe one segment register: selector, base, limit and access rights.

/*
 * Write the four VMCS fields that describe one guest segment register:
 * selector, base, limit and access rights.  seg is a SEG_* index from
 * vmx.h.  The VMCS field encodings are laid out so that segment i's
 * fields are at the ES field + 2*i (see vmx.h), which this function
 * relies on.  Unlike real segment registers, the VMCS holds the
 * "hidden" descriptor cache explicitly, so we must fill it in even
 * though no GDT exists in the guest yet (SDM 27.4.1).
 */
static void
guestseg(int seg, uint sel, uint64 base, uint limit, uint ar)
{
  vmwrite(GUEST_ES_SELECTOR + 2*seg, sel);
  vmwrite(GUEST_ES_BASE + 2*seg, base);
  vmwrite(GUEST_ES_LIMIT + 2*seg, limit);
  vmwrite(GUEST_ES_AR_BYTES + 2*seg, ar);
}

The VMCS field encodings for the eight segment registers are laid out so that segment number seg is 2*seg after the ES field. vmx.h defines the numbers: SEG_ES, SEG_CS, SEG_SS, SEG_DS, SEG_FS, SEG_GS, SEG_LDTR, SEG_TR. Note that the order is the VMCS order (ES, CS, SS, DS, FS, GS), not the x86 segment-register encoding order.

One thing is new compared with real segment registers. In protected mode, loading a selector makes the CPU read a descriptor from the GDT and cache its base, limit and access rights in hidden registers. The VMCS stores those hidden parts explicitly. Our guest starts with no GDT at all, so the CPU cannot look anything up: we must fill in the cache ourselves (SDM 27.4.1).

Step 1: Declarations and the TRUE capability MSRs

SDM: Appendix A.1 “Basic VMX Information” (bit 55 of IA32_VMX_BASIC); A.2 “Reserved Controls and Default Settings”.

Start vmcsinit() with:

  uint pin, cpu, cpu2, exitctl, entryctl;
  /*
   * IA32_VMX_BASIC bit 55 says that the "TRUE" capability MSRs exist.
   * They report the real default-1 controls; the older MSRs force
   * some bits (like CR3-load exiting) to 1 on old CPUs.
   */
  int t = (rdmsr(MSR_IA32_VMX_BASIC) & VMX_BASIC_TRUE_CTLS) != 0;
  uint64 cr0, cr4;
  int seg;

pin, cpu, cpu2, exitctl and entryctl will hold the five 32-bit control words (pin-based, primary processor-based, secondary processor-based, VM-exit, VM-entry) that we compute in Step 3. cr0 and cr4 are for the guest control registers (Step 9), and seg is a loop variable (Step 10).

The interesting line is t. Many control bits were reserved in the first VT-x processors and had to be 1 (“default1” controls). Later processors let some of them be 0. A CPU announces that by setting bit 55 of IA32_VMX_BASIC, and then provides four extra MSRs, the TRUE_* ones, that report the real allowed settings. If you used the old MSR on such a CPU, it would report “must be 1” for a bit that you are actually free to clear. Step 3 uses t to choose between the two.

Only the secondary controls have no TRUE_ MSR (A.3.3): none of those bits is default1.

Step 2: adjustctl(), compute a legal control word

SDM: Appendix A.3.1 to A.3.3, A.4.1 and A.5 (the allowed-0 and allowed-1 halves of each capability MSR).

This is the first function you write. It goes above vmcsinit(); its signature is given:

/*
 * Compute a legal value for a VM-execution/exit/entry control field.
 *
 * Each control capability MSR (SDM Appendix A.3-A.5) has two halves:
 * the low 32 bits are the "allowed-0 settings" (a bit that is 1 here
 * must be 1 in the control: the CPU requires it), and the high 32 bits
 * are the "allowed-1 settings" (a bit that is 0 here must be 0 in the
 * control: the CPU doesn't support it).
 *
 * Combine the controls we want with those the CPU requires (the
 * allowed-0 settings, in the low half of the capability MSR), and
 * check that the CPU allows all we want (allowed-1, the high half).
 *
 * msr is the capability MSR, want the bits we need, and *ctl receives
 * the value to write to the VMCS.  Returns 0, or -1 if some wanted
 * control is unsupported.  Called only by vmcsinit().
 */
static int
adjustctl(uint msr, uint want, uint *ctl)

and its body is:

  uint64 cap = rdmsr(msr);
  /* Force on the required bits, then mask off unsupported ones. */
  uint v = (want | (uint)cap) & (uint)(cap >> 32);

  /* If masking dropped a bit we asked for, the CPU can't do it. */
  if((v & want) != want){
    cprintf("vmx: controls 0x%x unsupported (MSR 0x%x)\n",
            (uint64)(want & ~v), (uint64)msr);
    return -1;
  }
  *ctl = v;
  return 0;

Every VM-execution, VM-exit and VM-entry control word is a 32-bit vector of bits, and the CPU has opinions about which bits may be 0 or 1. Each control word has a capability MSR, and the MSR holds two 32-bit halves:

• bits 31:0, “allowed-0 settings”. If bit X is 1 here, then control X must be 1. (It is not allowed to be 0.)
• bits 63:32, “allowed-1 settings”. If bit 32+X is 0 here, then control X must be 0. (It is not allowed to be 1.)

This is the same pattern as the CR0 and CR4 fixed bits in part 1. FIXED0 said “a 1 here must be 1”, which is the low half. FIXED1 said “a 0 here must be 0”, which is the high half. The legal value is again “OR in the low half, AND with the high half”, and that is exactly the expression for v.

The function then checks something the control registers did not need. We want certain controls (the argument want). If masking with the high half cleared one of them, the CPU does not support what we asked for, and we cannot make the VM work. We print which bits are missing and return -1. Note the order: forcing on bits that the CPU requires and that we did not ask for is fine (they were reserved or default1); only losing a bit we wanted is an error.

Pitfalls. rdmsr() returns 64 bits, so keep the casts: the low half is (uint)cap and the high half is (uint)(cap >> 32). And %x in cprintf reads a 64-bit argument, which is why the arguments are cast to uint64.

Step 3: Compute the five control words

SDM: 27.6.1 “Pin-Based VM-Execution Controls”, 27.6.2 “Processor-Based VM-Execution Controls”, 27.7.1 “VM-Exit Controls”, 27.8.1 “VM-Entry Controls”; Appendix A.3 to A.5.

  /*
   * Compute all five control words.  See SDM 27.6 (execution
   * controls), 27.7 (exit controls) and 27.8 (entry controls).
   */
  if(adjustctl(t ? MSR_IA32_VMX_TRUE_PINBASED_CTLS : MSR_IA32_VMX_PINBASED_CTLS,
               /*
                * Pin-based: exit on external interrupts (so the host
                * kernel, not the guest, handles them) and on NMIs.
                */
               PIN_EXTINT_EXITING | PIN_NMI_EXITING, &pin) < 0 ||
     adjustctl(t ? MSR_IA32_VMX_TRUE_PROCBASED_CTLS : MSR_IA32_VMX_PROCBASED_CTLS,
               /*
                * Primary processor-based: exit on HLT, exit on every
                * IN/OUT (no I/O bitmap), and enable the secondary
                * controls word, which holds EPT and unrestricted guest.
                */
               CPU_HLT_EXITING | CPU_UNCOND_IO_EXITING | CPU_ACTIVATE_SECONDARY,
               &cpu) < 0 ||
     adjustctl(MSR_IA32_VMX_PROCBASED_CTLS2,
               /*
                * Secondary: nested paging via EPT, and unrestricted
                * guest (lets the guest run with CR0.PE/PG off, i.e.
                * real or paging-less protected mode).  Unrestricted
                * guest requires EPT.
                */
               CPU2_ENABLE_EPT | CPU2_UNRESTRICTED_GUEST, &cpu2) < 0 ||
     adjustctl(t ? MSR_IA32_VMX_TRUE_EXIT_CTLS : MSR_IA32_VMX_EXIT_CTLS,
               /*
                * Exit: the host is a 64-bit kernel, so the processor
                * must return to 64-bit mode (HOST_ADDR_SPACE_SIZE);
                * save the guest's EFER into the VMCS on exit and load
                * the host's EFER from the VMCS.  "Acknowledge
                * interrupt on exit" is deliberately not set.
                */
               EXIT_HOST_ADDR_SPACE_SIZE | EXIT_SAVE_IA32_EFER |
               EXIT_LOAD_IA32_EFER, &exitctl) < 0 ||
     adjustctl(t ? MSR_IA32_VMX_TRUE_ENTRY_CTLS : MSR_IA32_VMX_ENTRY_CTLS,
               /*
                * Entry: load the guest's EFER from the VMCS.  The
                * "IA-32e mode guest" bit is set later, in vmrun(),
                * according to the guest's EFER.LMA.
                */
               ENTRY_LOAD_IA32_EFER, &entryctl) < 0)
    return -1;

This is one if with five calls to your adjustctl(), joined by ||: the first call that fails makes the whole function return -1 (the others are not tried). Each call picks the TRUE or the plain MSR with t from Step 1, except the secondary controls. Here is what each word asks for and why:

Unrestricted guest (SDM 28.6). The first VT-x processors required CR0.PE and CR0.PG to be 1 in VMX operation, so a guest could run only in paged protected mode, and a hypervisor had to emulate real mode and paging-less protected mode in software. Later processors have a VM-execution control, “unrestricted guest”, that lifts the restriction: the guest may run with CR0.PE or CR0.PG clear, and the CPU treats those bits as it does outside VMX operation. We use it so that our guest can start in 32-bit protected mode with paging off, which is exactly the state in which xv6’s own boot code starts, and then make its own way to 64-bit mode. The manual notes that this control requires “enable EPT”, which is why both are in the secondary word. Where the guest runs without paging, each guest linear address is passed straight to the EPT as a guest-physical address (28.6), which is just what our flat guest wants.

EFER on entry and exit. IA32_EFER holds the long-mode bits (LME and LMA). The guest’s trampoline sets LME and then turns paging on, at which point the CPU sets LMA. The host is already in long mode. If the CPU did not swap EFER, the guest’s value would leak into the host (or the other way round). So we ask the CPU to save the guest’s EFER into the VMCS at exit and to load the host’s EFER, and to load the guest’s EFER at entry. The “host address-space size” bit (SDM 27.7.1) tells the CPU that the host is in 64-bit mode after an exit: its value is loaded into CS.L, EFER.LME and EFER.LMA.

We do not set “acknowledge interrupt on exit”. An external interrupt that causes an exit therefore stays pending in the local APIC, and xv6 takes it normally through its trap path when it enables interrupts again.

The “IA-32e mode guest” entry control is not set here. It must agree with the guest’s EFER.LMA, which changes while the guest runs, so vmrun() recomputes it before every entry.

Step 4: Refuse a CPU that forces unwanted exits

SDM: 27.6.2; Appendix A.3.2 and A.2.

  /*
   * Without the TRUE capability MSRs these may be forced on; the
   * guest's trampoline would then exit on MOV to CR3.  Our run loop
   * has no code to emulate CR accesses or to build I/O or MSR bitmaps,
   * so fail rather than run a VM that would behave unexpectedly.
   */
  if(cpu & (CPU_CR3_LOAD_EXITING | CPU_CR3_STORE_EXITING |
            CPU_USE_IO_BITMAPS | CPU_USE_MSR_BITMAPS)){
    cprintf("vmx: CPU forces unwanted exiting controls 0x%x\n", (uint64)cpu);
    return -1;
  }

adjustctl() forces on every bit that the CPU requires. On an old CPU without TRUE MSRs, some of the default1 primary controls are bits we do not want: CR3-load exiting, CR3-store exiting, and the two bitmaps. If they are set, the guest’s MOV to CR3 would exit and we have no code to handle it, and a bitmap control without a bitmap address would be an invalid configuration. This check turns that into a clear error message at vmcreate() time instead of a mysterious exit later.

On a CPU with TRUE MSRs this check should not fire, because the TRUE MSRs report that those bits may be 0.

Step 5: Write the control fields

SDM: 27.6.1, 27.6.2, 27.7.1, 27.8.1; Appendix B for the field encodings.

  /*
   * Control fields.  No MSR bitmap, so every RDMSR/WRMSR exits;
   * unconditional I/O exiting, so every IN/OUT exits.
   */
  vmwrite(PIN_BASED_VM_EXEC_CONTROL, pin);
  vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu);
  vmwrite(SECONDARY_VM_EXEC_CONTROL, cpu2);
  vmwrite(VM_EXIT_CONTROLS, exitctl);
  vmwrite(VM_ENTRY_CONTROLS, entryctl);

Until now we only computed numbers. These five vmwrite() calls put them in the VMCS. Notice the comment: “no MSR bitmap” means that the primary control “use MSR bitmaps” is off, and the CPU then makes every RDMSR and WRMSR in the guest exit. That is what we want for now, since the guest writes EFER and the kernel emulates it.

Step 6: The exception bitmap

SDM: 27.6.3 “Exception Bitmap”; 28.2 “Other Causes of VM Exits”.

  /*
   * The exception bitmap has one bit per exception vector 0-31; a set
   * bit makes that exception cause a VM exit instead of being
   * delivered through the guest's IDT.  The guest has no IDT yet, so
   * without this any guest exception would escalate to a triple fault
   * with no information.  With it, each exception exits with its
   * vector, error code, and RIP.
   */
  vmwrite(EXCEPTION_BITMAP, 0xFFFFFFFF);  /* no guest IDT: exit on everything */
  /*
   * For page faults the mask/match pair further filters which error
   * codes exit; with all bits set in the bitmap, mask = match = 0
   * means "every page fault exits" (SDM 27.6.3).
   */
  vmwrite(PAGE_FAULT_ERROR_CODE_MASK, 0);
  vmwrite(PAGE_FAULT_ERROR_CODE_MATCH, 0);

When the guest takes an exception (a page fault, a general-protection fault, a divide error, …), the CPU normally delivers it through the guest’s IDT. Our guest starts with no IDT (its IDTR limit is 0, Step 11), so a guest bug would turn into a double fault and then into a triple fault, and the guest would simply die with no information about what happened. With a 1 in the exception bitmap for each vector, any exception becomes a VM exit that reports the vector, the error code and the guest RIP, and runvm prints them (runvm: guest exception 13, error code ...). That is a debuggable failure instead of a silent one. The bitmap is always 32 bits wide, one bit per vector 0 to 31.

Page faults have one extra filter (SDM 27.6.3): the bit for vector 14 is combined with the page-fault error code, the mask and the match fields. Setting both to 0 means “every page fault exits”, so the bitmap alone decides.

A guest in this assignment has paging off at the start, so page faults only happen after the guest builds its own page tables. If one occurs, you want to see it.

Step 7: Fields that disable optional features

SDM: 27.6.7 “CR3-Target Controls”; 27.7.2 and 27.8.2 (MSR save and load areas); 27.8.3 (event injection); 27.6.11 “Extended-Page-Table Pointer”.

  /* No CR3-target values (they would let some MOV to CR3 skip exiting). */
  vmwrite(CR3_TARGET_COUNT, 0);
  /* No MSR save/load lists on exit or entry. */
  vmwrite(VM_EXIT_MSR_STORE_COUNT, 0);
  vmwrite(VM_EXIT_MSR_LOAD_COUNT, 0);
  vmwrite(VM_ENTRY_MSR_LOAD_COUNT, 0);
  /* Don't inject any event (interrupt/exception) on VM entry. */
  vmwrite(VM_ENTRY_INTR_INFO_FIELD, 0);
  /* Tell the processor where this guest's EPT lives (SDM 27.6.11). */
  vmwrite(EPT_POINTER, vm->eptp);

The VMCS contents are undefined when you start (the CPU only sets what you VMWRITE), so every field that the CPU is going to read has to be written, even when “off” is the right value. This block zeroes the count fields that would otherwise be random garbage: the number of CR3 targets, the three MSR-list lengths (we use EFER save and load controls from Step 3 instead of lists), and the event-injection field (bit 31, the “valid” bit, is 0, so nothing is injected).

The last line is the one that connects this part to part 2: it tells the CPU where the EPT is. vm->eptp is the physical address of the EPT PML4 ORed with the memory type (write-back) and the page-walk length, built for you in vmcreate(). The format of the EPTP is Table 27-9 in SDM 27.6.11.

Step 8: Control register ownership and the host return address

SDM: 27.6.6 “Guest/Host Masks and Read Shadows for CR0 and CR4”; 26.8; 27.5 “Host-State Area”.

  /*
   * The guest/host mask says which CR0/CR4 bits are owned by the host:
   * guest writes to an owned bit that differ from the read shadow
   * cause an exit, and guest reads of an owned bit return the shadow
   * value.  The host owns CR4.VMXE: the guest reads it as 0 and can't
   * change it.  (The real CR4.VMXE must stay 1 while in VMX
   * operation, see GUEST_CR4 below.)  CR0 has no owned bits, so the
   * guest may freely change CR0.PG, CR0.PE etc.
   */
  vmwrite(CR0_GUEST_HOST_MASK, 0);
  vmwrite(CR0_READ_SHADOW, 0);
  vmwrite(CR4_GUEST_HOST_MASK, CR4_VMXE);
  vmwrite(CR4_READ_SHADOW, 0);

For CR0 and CR4 the VMCS has two extra fields each: a guest/host mask and a read shadow. A 1 in the mask means the host “owns” that bit:

• if the guest reads the register (MOV from CR), the owned bits come from the read shadow instead of the real register;
• if the guest tries to write a value that differs from the shadow in an owned bit, the write causes a VM exit.

A 0 in the mask means the guest owns the bit: reads and writes go straight to the real register.

We make one bit host-owned, CR4.VMXE. Remember from part 1 that the real CR4.VMXE must be 1 for as long as the CPU is in VMX operation. A guest, though, should think it is on a machine without VMX, so we give it a shadow of 0. As a result, the guest reads CR4.VMXE as 0 and, if it tries to set it, it exits. (Our guest never does.) For CR0 we own nothing, so the guest can toggle PE and PG, which its trampoline must do.

  /*
   * Where the processor resumes the host after every VM exit.  This
   * never changes, so it is written once here.  (HOST_RSP is written
   * by vmx_enter on every entry, since it depends on the stack depth.)
   */
  vmwrite(HOST_RIP, (uint64)vmx_exit);

On a VM exit the CPU has to know where to resume the host, so it loads the host state from the VMCS: RIP, RSP, CR0, CR3, CR4, the segment selectors and so on (SDM 27.5). HOST_RIP is the address where the host resumes after every exit. It is always the same label, vmx_exit, which lives in vmxasm.S (you write it in part 4). Because it never changes, we write it once, here. In part 4 you will see that the other host fields do change from one entry to the next (CR3 depends on the process, TR on the CPU) and are rewritten by vmxhoststate() before every entry.

Note that vmx_exit is declared as extern char vmx_exit[], so the cast (uint64)vmx_exit is its address.

Step 9: Guest control registers

SDM: 26.8 “Restrictions on VMX Operation”; 29.3.1.1 “Checks on Guest Control Registers, Debug Registers, and MSRs”; Appendix A.7 and A.8.

  /*
   * Guest state: flat 32-bit protected mode with paging off.
   * Unrestricted guest allows CR0.PE and CR0.PG to differ from the
   * fixed-bit MSRs; every other fixed bit still applies.
   * So start from PE|ET|NE, force on any other FIXED0 bits (except
   * PE and PG), and mask with FIXED1.  NE is required by FIXED0.
   */
  cr0 = CR0_PE | CR0_ET | CR0_NE;
  cr0 |= rdmsr(MSR_IA32_VMX_CR0_FIXED0) & ~(uint64)(CR0_PE | CR0_PG);
  cr0 &= rdmsr(MSR_IA32_VMX_CR0_FIXED1);

At VM entry the CPU checks that the guest’s CR0 is legal in VMX operation, with exactly the fixed-bit rule of part 1 (SDM 29.3.1.1: “the CR0 field must not set any bit to a value not supported in VMX operation”). The exception is that with unrestricted guest, bit 0 (PE) and bit 31 (PG) are not checked. That is why we cannot just copy the part 1 formula: FIXED0 says PE and PG must be 1, and we want them off. So we OR in all the FIXED0 bits except those two, then AND with FIXED1.

We start from PE | ET | NE:

• PE = 1: protected mode (it is the point of the exercise);
• PG = 0: paging off, so a guest linear address equals its guest-physical address and goes straight to the EPT (SDM 28.6);
• NE = 1: FIXED0 requires it anyway. (ET is bit 4 of CR0; we set it as the original code does.)

  /*
   * CR4 in the VMCS must satisfy the fixed bits too, and since the CPU
   * is in VMX operation, VMXE must be 1 in the real CR4; the guest
   * sees it as 0 through the read shadow set above.
   */
  cr4 = CR4_VMXE | rdmsr(MSR_IA32_VMX_CR4_FIXED0);
  cr4 &= rdmsr(MSR_IA32_VMX_CR4_FIXED1);
  vmwrite(GUEST_CR0, cr0);
  /* CR3 is irrelevant until the guest turns paging on. */
  vmwrite(GUEST_CR3, 0);
  vmwrite(GUEST_CR4, cr4);
  /* DR7 reset value (bit 10 is reserved-1): no hardware breakpoints. */
  vmwrite(GUEST_DR7, 0x400);

For CR4 there are no exceptions: the full part 1 formula applies. Since VMX is on, VMXE is in FIXED0 anyway, but we OR it in explicitly, as we did in part 1. This is the real CR4 the guest runs with. The guest cannot see it, thanks to the shadow from the previous step.

CR3 is 0 because paging is off; the guest loads its own CR3 before it turns paging on. DR7 is the debug control register. We write 0x400, its reset value (bit 10 is reserved-1), so that no hardware breakpoint is enabled.

Step 10: Guest segment registers

SDM: 27.4.1 “Guest Register State” (Table 27-2, the access-rights format); 29.3.1.2 “Checks on Guest Segment Registers”; 29.3.1.3.

  /*
   * Segments (SDM 27.4.1).  CS is a flat 32-bit code segment with
   * selector 0x08, and ES/SS/DS/FS/GS are flat 32-bit data segments
   * with selector 0x10; base 0, limit 4 GB.  The selector values are
   * only labels here since the guest has no GDT, but they match the
   * layout the guest's own GDT will use.
   */
  guestseg(SEG_CS, 0x08, 0, 0xFFFFFFFF, AR_CODE32);
  for(seg = SEG_ES; seg <= SEG_GS; seg++)
    if(seg != SEG_CS)
      guestseg(seg, 0x10, 0, 0xFFFFFFFF, AR_DATA32);
  /*
   * LDTR is marked unusable (as after reset, no LDT), and TR must be
   * valid for VM entry: a busy TSS (32-bit, limit 0xFFFF) is the
   * simplest value that passes the guest-state checks of SDM 29.3.1.2.
   */
  guestseg(SEG_LDTR, 0, 0, 0, AR_UNUSABLE);
  guestseg(SEG_TR, 0, 0, 0xFFFF, AR_TSS32_BUSY);

This block comes right after the GUEST_DR7 line. It describes the segment registers of a CPU that has just started in flat 32-bit protected mode, as if a bootloader had loaded a GDT with a flat code segment and a flat data segment:

One call sets CS, a loop sets the five data segments (skipping CS, which was just done), and one call each handles LDTR and TR. The loop variable seg was declared in Step 1 for this purpose. The segment indices are the SEG_* constants, not the x86 encodings of the segment registers.

The selector values are only labels, because the guest has no GDT yet. We pick 0x08 and 0x10 because that is the layout of the GDT the guest’s own trampoline loads later.

Reading the access-rights values. Table 27-2 in SDM 27.4.1 gives the layout of the 32-bit access-rights field: bits 3:0 type, bit 4 S, bits 6:5 DPL, bit 7 P, bit 14 D/B, bit 15 G, bit 16 unusable. vmx.h defines the four values used above (AR_CODE32, AR_DATA32, AR_TSS32_BUSY, AR_UNUSABLE) and repeats the layout in a comment. It is worth decoding each one bit by bit and comparing it with the table. The checks in SDM 29.3.1.2 tell you what the CPU will accept:

• CS needs P = 1, S = 1 and a code type that is accessed (with unrestricted guest, type 3 would also be allowed; we use real code);
• a segment with a 4 GB limit needs G = 1, since a limit above 1 MB must be expressed in 4 KB units;
• TR must be usable and a busy TSS (type 11 for a 32-bit guest not in IA-32e mode), with S = 0 and P = 1;
• LDTR may be unusable, and if it is, its other fields are not checked.

Do not forget TR. A guest with no valid TR fails the entry checks, even though our guest never uses a task switch.

How a mistake here shows up. The mistakes in this block are not detected when you write them; the CPU checks the guest state only at VM entry. After part 4, when vmrun() can enter the guest, a wrong value here shows up in runvm as:

runvm: VM entry failed, reason 33, qualification 0x0

Reason 33 is “VM-entry failure due to invalid guest state” (SDM 29.8 and Appendix C): the entry failed one of the checks of SDM 29.3.1. The qualification is 0 for most causes, so it will not tell you which segment field is wrong; compare each field with SDM 29.3.1.2. (If you want to try before then, see the read-back checkpoint below.)

Step 11: Descriptor tables

SDM: 27.4.1; 29.3.1.3 “Checks on Guest Descriptor-Table Registers”.

  /* The guest has no GDT or IDT until it loads its own. */
  vmwrite(GUEST_GDTR_BASE, 0);
  vmwrite(GUEST_GDTR_LIMIT, 0);
  vmwrite(GUEST_IDTR_BASE, 0);
  vmwrite(GUEST_IDTR_LIMIT, 0);

GDTR and IDTR have only a base and a limit (no selector, no access rights). The guest loads its own GDT in its trampoline. The IDT stays empty, which is why Step 6 made every exception a VM exit.

Step 12: The rest of the guest state

SDM: 27.4.2 “Guest Non-Register State”; 29.3.1.5 “Checks on Guest Non-Register State”; 27.4.1.

  /*
   * Non-register guest state (SDM 27.4.2).  Interruptibility 0 means
   * no blocking by STI/MOV SS; activity state 0 means "active" (not
   * halted).  Pending debug exceptions must be consistent with that.
   */
  vmwrite(GUEST_INTERRUPTIBILITY_INFO, 0);
  vmwrite(GUEST_ACTIVITY_STATE, 0);
  vmwrite(GUEST_PENDING_DBG_EXCEPTIONS, 0);
  /* No shadow VMCS: the link pointer must be all ones. */
  vmwrite(VMCS_LINK_POINTER, ~0ULL);
  /* Remaining MSR-backed guest state: all zero. */
  vmwrite(GUEST_IA32_DEBUGCTL, 0);
  vmwrite(GUEST_SYSENTER_CS, 0);
  vmwrite(GUEST_SYSENTER_ESP, 0);
  vmwrite(GUEST_SYSENTER_EIP, 0);
  return 0;

These fields do not correspond to a register that software can read; they describe what the CPU was in the middle of when the guest was stopped.

• Interruptibility state records “blocking” conditions, such as the one-instruction shadow after STI or MOV SS. 0 means the guest can take an interrupt at once.
• Activity state 0 means the CPU is “active” (executing). The other values are HLT, shutdown and wait-for-SIPI.
• Pending debug exceptions must be consistent with the other two. Zero is.
• VMCS link pointer. This field is used only with VMCS shadowing, which we do not use. The manual says to set it to all ones to avoid entry failures (SDM 27.4.2). If you leave a 0 there (the VMCS memory starts out zeroed), the CPU will treat it as a pointer to another VMCS, find no valid VMCS there, and refuse the entry (SDM 29.3.1.5; exit reason 33, most likely with qualification 4, see 29.8).
• The MSR-backed fields (DEBUGCTL, the three SYSENTER registers) are loaded into real MSRs at entry. Zero is a valid value.

Finally return 0 reports success to vmcreate().

RIP, RSP, RFLAGS and EFER are not written in this function. The guest state area also has fields for them, but their values change while the guest runs and are kept in struct vm between runs, so vmrun() copies them into the VMCS before every entry (part 5). vmcreate() sets the initial vm->rflags to 2 (bit 1 of RFLAGS always reads 1), and runvm sets the initial RIP with vmsetreg().

Checkpoint

There are two ways to tell that vmcsinit() works, depending on whether part 4 exists yet.

Check 1: before part 4 (the vmcreate() check). Build and boot, then run

$ runvm helloguest

runvm calls vmcreate() first (see its main()), and vmcreate() calls vmcsinit(). What you see tells you how far you got:

• If you see runvm: vmcreate failed, vmcsinit() returned -1 or something earlier in vmcreate() failed. The kernel prints a line just before it. vmx: controls 0x... unsupported (MSR 0x...) came from adjustctl(): the first number is the bits the CPU refused, the second is the MSR (for instance 0x48B is IA32_VMX_PROCBASED_CTLS2, so a missing EPT or unrestricted-guest feature).
• If the kernel panics after printing vmwrite 0x... = 0x...: error N, one of your vmwrite() calls failed. Look up N in SDM Table 33-1 (33.4): 12 is “unsupported VMCS component” (a bad field encoding), 13 is “write to a read-only component”.
• If neither happens, vmcreate() returned a VM id and runvm went on to vmsetreg() and vmrun(). (What vmrun() does without part 4 is not a bug of this part: you will see that in the next part.)

To see that the values are right, add a temporary check at the end of vmcsinit(), just before return 0; (remove it afterwards):

  cprintf("vmcs: cpu2=0x%x cr0=0x%x cr4=0x%x link=0x%x\n",
          vmread(SECONDARY_VM_EXEC_CONTROL), vmread(GUEST_CR0),
          vmread(GUEST_CR4), vmread(VMCS_LINK_POINTER));

The VMCS is current at this point, so VMREAD works. Expect:

• cpu2=0x82: bit 1 (EPT) and bit 7 (unrestricted guest), and nothing else. For the secondary controls the allowed-0 half of the MSR is always 0 (SDM A.3.3), so no other bit is forced on.
• cr0: bit 0 (PE) set, bit 5 (NE) set, bit 31 (PG) clear. The exact value can have more bits on other CPUs, but PG must be 0.
• cr4: bit 13 (VMXE) set.
• link=0xffffffffffffffff (all ones).

Check 2: after part 4 (the VM entry check). Once vmx_enter() and vmxhoststate() exist, runvm helloguest actually tries to enter the guest. Then the VMCS you wrote is checked by the hardware, and the result is printed by runvm:

Optional experiment. After part 4, make the TR a segment that is AR_UNUSABLE instead of a busy TSS and boot. Which line of SDM 29.3.1.2 does the entry violate? (It is in the TR checks: “the unusable bit must be 0”.)

If something fails:

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. In adjustctl(), why is it enough to compute (want | low) & high? What does the function do in the case where want has a bit that high has cleared, and why is returning an error better than silently dropping the bit?
2. The exception bitmap is all ones. If it were 0, what would happen to a page fault in the guest, step by step, given that the guest’s IDTR limit is 0? Why is the all-ones setting more useful to someone debugging a guest?
3. Why must “enable EPT” be 1 whenever “unrestricted guest” is 1? Think about what the CPU does with a guest linear address when CR0.PG = 0 (SDM 28.6).
4. The real CR4 in the VMCS has VMXE = 1 but the guest reads it as 0. Which two VMCS fields make that work? What happens if the guest tries to set CR4.VMXE to 1, and why do we not let it?

Part 4: Entering and leaving the guest

Parts 1 to 3 gave you a CPU in VMX operation and a fully configured VMCS. In this part you make the CPU actually run the guest, and you make it come back. When you finish, the guest runs its first few instructions, executes RDMSR, and the CPU returns control to your kernel code with the exit reason in the VMCS. Your vmrun() loop turns that into a report, and runvm prints it.

Two pieces of code do the work:

• vmxhoststate() in vmx.c tells the CPU what state to load when the guest exits. You write it in four steps.
• vmx_enter and vmx_exit in vmxasm.S switch between the host and the guest. This is assembly, and it is the trickiest code in the homework. You write two parts of it; the rest we give you.

Where to look things up. As before, “SDM” is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026). Section numbers are for that edition.

What is given and what you write

The vmxhoststate() steps below give you the code to paste. The two assembly blocks are Your turn sections: we describe the goal and give hints, but you write the instructions yourself.

Section 1: vmxhoststate()

When the guest exits, the CPU must get back to a state where your kernel can run. It does not remember what that state was. It loads it from the host-state area of the VMCS, which you fill in. This is the mirror image of the guest-state area: the guest-state area is loaded on entry, the host-state area on exit.

Why write it before every entry? You could imagine filling this area once, in vmcsinit(), like HOST_RIP. That would be a bug, for two reasons.

• It depends on the CPU. The host state names this CPU’s TSS and GDT. After a VMCLEAR the process can be rescheduled on a different CPU (see the invariant in vmrun()), which has a different TSS and GDT.
• It depends on the process. CR3 is the page table of the process that called vmrun(). Each process has its own. If you left another process’s CR3 in the VMCS, the exit would land in a page table that does not map this process’s kernel stack.

So vmrun() calls vmxhoststate() on every iteration, with interrupts off and the VMCS current. Open vmx.c, find the empty body, and paste the blocks below in order.

Step 1: Control registers

SDM: 27.5 “Host-State Area”; 30.5.1 “Loading Host Control Registers, Debug Registers, MSRs”; 29.2.2 “Checks on Host Control Registers, MSRs, and SSP”.

  struct cpu *c = mycpu();

  /* Control registers: the kernel's own, so exits land in the kernel. */
  vmwrite(HOST_CR0, rcr0());
  /* The current process's page table; the kernel half is shared. */
  vmwrite(HOST_CR3, rcr3());
  vmwrite(HOST_CR4, rcr4());

c is the CPU we are running on; Step 3 uses it. The three vmwrites copy the CPU’s current CR0, CR3 and CR4 into the host-state area. On exit, the CPU loads them back, so the kernel finds itself in the same address space and with the same paging mode as before the entry.

We copy the live values rather than computing new ones. That is automatically legal: the CPU checks at entry that the host CR0 and CR4 obey the fixed-bit rules (the same rules as in part 1, see 29.2.2), and the live registers already do, because vmxinit() made them so. One detail from 30.5.1: a few CR0 bits are never changed by an exit (for example ET, CD and NW), so they are not checked either.

CR4 includes VMXE, which must stay 1 while we are in VMX operation. Copying the live value keeps it.

Step 2: Selectors

SDM: 27.5; 29.2.3 “Checks on Host Segment and Descriptor-Table Registers”; 30.5.2 “Loading Host Segment and Descriptor-Table Registers”.

  /*
   * Selectors.  Host selectors must have RPL = 0 and TI = 0, and CS
   * and TR must be nonzero.  CS is the kernel code segment; the data
   * segment registers are not used in 64-bit mode so a null selector
   * is fine (allowed since the host is 64-bit).
   */
  vmwrite(HOST_CS_SELECTOR, SEG_KCODE << 3);
  vmwrite(HOST_SS_SELECTOR, 0);
  vmwrite(HOST_DS_SELECTOR, 0);
  vmwrite(HOST_ES_SELECTOR, 0);
  vmwrite(HOST_FS_SELECTOR, 0);
  vmwrite(HOST_GS_SELECTOR, 0);
  vmwrite(HOST_TR_SELECTOR, SEG_TSS << 3);

The host-state area has a selector field for each of CS, SS, DS, ES, FS, GS and TR. (There is none for LDTR; the exit simply makes it unusable.) The CPU checks them at entry, and the rules (29.2.3) are:

• RPL (bits 1:0) and TI (bit 2) must be 0 in every selector;
• CS and TR must not be 0;
• SS may be 0 only when the “host address-space size” exit control is 1, as it is for us (30.5.2 says the same from the exit side: SS can be zero only if the exit is to 64-bit mode).

SEG_KCODE << 3 is the kernel code selector: the GDT index shifted left by 3, with RPL 0 and TI 0. SEG_TSS << 3 is the selector of the TSS descriptor in this CPU’s GDT. The data selectors are 0. In 64-bit mode the data segment registers are not used for address translation, so a null selector is enough, and the kernel loads real ones later if it needs them.

The base and limit of CS, and the type of TR, are not in the VMCS. The exit fills them with fixed values (30.5.2: CS base 0 and limit FFFFFFFF, TR limit 67, and so on). That is why the selectors alone are enough for CS.

Step 3: Bases of FS, GS, TR, GDTR and IDTR

SDM: 27.5; 29.2.3 (bases must be canonical); 30.5.2.

  /*
   * Segment bases.  FS/GS bases live in MSRs, and a VM exit loads them
   * from these fields, so we write back the current values and the
   * exit leaves them unchanged.  (xv6-64 does not use them for per-CPU
   * data; mycpu() finds the CPU from its local APIC id.)  The exit
   * does not read the TR, GDTR or IDTR from memory; it takes their
   * bases from these fields, so they must name this CPU's TSS and GDT
   * and the kernel's IDT.
   */
  vmwrite(HOST_FS_BASE, rdmsr(MSR_IA32_FS_BASE));
  vmwrite(HOST_GS_BASE, rdmsr(MSR_IA32_GS_BASE));
  vmwrite(HOST_TR_BASE, (uint64)&c->ts);
  vmwrite(HOST_GDTR_BASE, (uint64)c->gdt);
  vmwrite(HOST_IDTR_BASE, (uint64)idt);

In 64-bit mode, the bases of FS and GS are not part of a descriptor; they live in two MSRs. The exit loads them from the VMCS, so we read the current values and store them, and the kernel sees the same bases after the exit as before.

TR is different. A selector alone does not tell the CPU where the TSS is, so the exit takes the base straight from HOST_TR_BASE (30.5.2: TR is “loaded from the host-state area”). It never reads the GDT to find it. Same for GDTR and IDTR: the CPU loads the bases from these fields, and sets both limits to FFFFH (30.5.2).

This is where “per CPU” bites. &c->ts is this CPU’s TSS and c->gdt is this CPU’s GDT, so they change when the process moves between CPUs. idt is the one kernel IDT, shared by all CPUs. All five bases must be canonical addresses (29.2.3); kernel addresses are.

A pitfall. HOST_TR_BASE must be a pointer to the TSS structure itself, not to something that contains it. In struct cpu, ts is a member, so &c->ts is what you want.

Step 4: SYSENTER MSRs and EFER

SDM: 27.5; 30.5.1; 27.7 “VM-Exit Control Fields” (the “load IA32_EFER” control).

  /* The kernel does not use SYSENTER; load zeros on exit. */
  vmwrite(HOST_IA32_SYSENTER_CS, 0);
  vmwrite(HOST_IA32_SYSENTER_ESP, 0);
  vmwrite(HOST_IA32_SYSENTER_EIP, 0);
  /* Loaded on exit because we set EXIT_LOAD_IA32_EFER in vmcsinit(). */
  vmwrite(HOST_IA32_EFER, rdmsr(EFER_MSR));
}

The SYSENTER fields exist in every VMCS, and the CPU always loads them on exit (30.5.1). xv6 does not use SYSENTER, so we write zeros. (The SYSENTER ESP and EIP fields are checked for canonical addresses, 29.2.2; zero is canonical.)

HOST_IA32_EFER is different: the CPU loads it only if the “load IA32_EFER” exit control is 1, which your part 3 code set. The value must be one that keeps the host in long mode: the check at 29.2.2 requires the LMA and LME bits of the field to equal the “host address-space size” exit control. Copying the live EFER satisfies this, because the kernel is in long mode and that control is 1.

The closing brace belongs to the function. The whole of vmxhoststate() is done.

Section 2: How the CPU gets into the guest and back

Before you write any assembly, understand the control flow. It is unusual, so read this section twice.

A function that returns when the guest exits

vmrun() calls vmx_enter(&vm->regs, 0) like any C function. It expects the call to return an int: 0 after a VM exit, -1 if the entry itself failed. But in between, the guest ran, and the guest does not return to its caller like a function does. How does this work?

The trick is that we choose where the CPU lands after an exit. The host-state area contains two more fields, HOST_RIP and HOST_RSP, and on every exit the CPU loads RIP and RSP from them (30.5.3). So:

• HOST_RIP is set to the address of the label vmx_exit. You did this in vmcsinit() (part 3). It never changes, so it is written once.
• HOST_RSP is set by vmx_enter, on every entry, to the value of %rsp at that moment.

Nobody calls vmx_exit. It is a landing pad. When the exit happens, the CPU is at vmx_exit with %rsp exactly as vmx_enter left it, pointing at the frame vmx_enter built. So vmx_exit can pop that frame and execute ret. That ret pops the return address of the original call, so it returns from vmx_enter, to vmrun(). From C’s point of view, vmx_enter called once and returned once, and the guest ran in the middle.

Why does HOST_RSP have to be written on every entry and not once? It holds a stack address, and the stack depends on which process is calling and how deep in the kernel it is. Different processes have different kernel stacks, so the value from the last entry is wrong for this one.

What the CPU saves and what it does not

On entry the CPU loads, from the guest-state area, the guest’s RSP, RIP and RFLAGS, along with control registers, segments and a few MSRs. On exit it stores them back (27.4.1 lists the registers that have VMCS fields: the control registers, DR7, RSP, RIP and RFLAGS, and the segment state).

It does not save or load the other 15 general-purpose registers (RAX, RBX, … R15). On exit they still hold the guest’s values; on entry they have to hold the guest’s values already. Software must copy them to and from memory. This is the reason vmx_enter and vmx_exit are assembly: C code between the register loads and the VMLAUNCH could use those registers for its own purposes.

Where do we keep them? In struct guestregs (vmx.h), which is the field vm->regs:

Each field is 8 bytes. The assembly uses these numbers directly, as in movq 16(%rdi), %rcx, so the structure and the assembly must agree. Notice two things:

• rdi is at offset 40, between rsi and rbp. It is not in alphabetical or numerical order. The order comes from the VMREG_* numbering in vmxapi.h. If you write offsets from memory of the x86 register order, you will get it wrong.
• There is no rsp field. The guest’s RSP is in the VMCS, not in this structure. Question 1 asks you why.

VMLAUNCH, VMRESUME and VMCLEAR

SDM: 27.1 “Overview” (launch state); 27.11.3 “Initializing a VMCS” (the VMLAUNCH and VMRESUME usage rules); 29.1 “Basic VM-Entry Checks”; 33.3, “VMLAUNCH/VMRESUME”.

Each VMCS has a launch state, which is either “clear” or “launched”. The two entry instructions differ in which state they require:

• VMLAUNCH requires “clear”, and on success changes the state to “launched”;
• VMRESUME requires “launched”.

Using the wrong one fails with a VM-instruction error (error 4, “VMLAUNCH with non-clear VMCS”, or error 5, “VMRESUME with non-launched VMCS”; see 33.4). The SDM’s recommended usage is: VMLAUNCH for the first entry after VMCLEAR, VMRESUME for later entries. VMCLEAR always sets the state back to “clear”.

vmrun() does a VMCLEAR after every exit, because the process may move to another CPU when interrupts are re-enabled, and a VMCS must be cleared on the CPU where it was used before it is used on another (27.11.1; part 3 explains this). So the VMCS is always “clear” at the next entry, and the design always passes launched = 0 and uses VMLAUNCH. The VMRESUME path exists in vmx_enter only for completeness; nothing here takes it. (It is slower than keeping the VMCS loaded, as the SDM notes at 27.11.3. We choose simplicity. The README discusses the trade-off.)

How an entry failure is reported

If VMLAUNCH works, it does not fall through: the next instruction executed is in the guest. So if the instruction after VMLAUNCH ever runs, the entry failed. The CPU reports this with the flags (33.2, “Conventions”):

The checks that fail this way (29.1, 29.2) are: no current VMCS, wrong launch state, invalid control fields (error 7) and invalid host-state fields (error 8). The error numbers are listed in 33.4. Your vmxhoststate() is the usual cause of error 8.

Entry can also fail later, in the checks of the guest state (29.3). Then the CPU does not fall through. Instead it loads the host state, as for an exit, and jumps to HOST_RIP, with bit 31 of the exit reason set (29.8; EXIT_REASON_ENTRY_FAILURE in vmx.h). So vmx_exit runs, and vmrun() sees it as an ordinary return with e.entryfail set.

The given code does not look at the flags. Reaching the instruction after VMLAUNCH is failure enough: vmx_enter returns -1, and vmrun() reads VM_INSTRUCTION_ERROR for the cause.

What the CPU does to the host on an exit

For a clear picture of the state at vmx_exit, here is what has happened by then (30.5):

• CR0, CR3, CR4, the segment registers, GDTR, IDTR, TR, and EFER are loaded from your host-state area;
• RIP and RSP come from HOST_RIP and HOST_RSP;
• RFLAGS is cleared except for bit 1, which is always set (30.5.3), so interrupts are off and nothing can interrupt the code at vmx_exit;
• the general-purpose registers are untouched: they still hold the guest’s values.

That last point is the whole difficulty of vmx_exit. All 15 registers are full of guest data, and you need at least one to hold a pointer to regs.

Section 3: The assembly

Open vmxasm.S. Near the top there is the constant HOST_RSP (0x6C14, the VMCS encoding of the field), then vmx_enter, then vmx_exit. Your two stripped functions look like this:

.globl vmx_enter
vmx_enter:
  # YOUR CODE, block A
  #   example, one line of it:  pushq %rbp   (saves the host's %rbp)

  testl %esi, %esi          # given (Step 3 below)

  # YOUR CODE, block B
  #   example, one line of it:  movq 8(%rdi), %rbx
  #   (%rbx = the 8 bytes at address regs + 8: the guest's rbx)

  jnz   1f                  # given: the launch and failure path
  ...

.globl vmx_exit
vmx_exit:
  # YOUR CODE
  #   example, one line of it:  movq %rbx, 8(%rdi)
  #   (the 8 bytes at address regs + 8 = the guest's rbx, if %rdi = regs)

On entry to vmx_enter, the System V AMD64 calling convention applies: %rdi holds regs (the first argument), %esi holds launched (the second), and the return address is at (%rsp). The callee-saved registers are %rbx, %rbp and %r12 to %r15. A C function must return with all of them unchanged. The guest will overwrite all of them, so we must save them before entering the guest and restore them when we come back.

Your turn, block A: save the host and write HOST_RSP

SDM: 27.5; 30.5.3 “Loading Host RIP, RSP, RFLAGS, and SSP”; 33.3, “VMWRITE–Write Field to Virtual-Machine Control Structure” (operand order and failure flags).

Goal. Build a frame on the host stack that vmx_exit can unwind, and tell the CPU to put %rsp at that frame after the next exit.

What to do, in order:

1. Push the six callee-saved registers.
2. Push regs (the value of %rdi) as the last push, so it is on top of the stack. vmx_exit will find it at (%rsp).
3. Write the current %rsp into the VMCS field HOST_RSP.

The frame you build must look like this (addresses grow downward; every slot is 8 bytes). The given failure path and the vmx_exit you write both depend on this layout:

Hints.

• One line of what you need, to show the syntax: pushq %rbp pushes the host’s %rbp onto the stack and moves %rsp down by 8. Repeat for the others.
• The offsets in the diagram follow from the push order: the first push lands highest. Work out which register you must push first to get %rbp at 48.
• HOST_RSP is written after all the pushes, never before. If you write it earlier, the exit lands at a stack that is shallower than your frame, and the unwinding in vmx_exit reads the wrong slots.
• VMWRITE takes two register operands: the field encoding in one register and the value in another. You cannot give it an immediate. The encoding is already defined for you as HOST_RSP in the assembly file. Load it into a scratch register such as %rax (we are going to overwrite %rax with the guest value anyway). In AT&T syntax the source (the value) comes first and the destination (the field encoding register) second, so check the order against the comment in the SDM’s operand description.
• Do not touch %rdi or %esi in this part. You still need %rdi as the pointer to regs, and %esi is tested right after your code.
• VMWRITE can fail, for example with no current VMCS. Here vmrun() has already executed VMPTRLD, so you do not need to check.

Given: test launched

  testl %esi, %esi

This sets ZF = 1 if launched == 0. We do it here, before the guest’s registers are loaded, because loading the guest’s %rsi destroys launched. The flags survive until the jnz below because mov does not change them. This is a constraint on your block B: use only mov between this testl and the jnz. An add, sub, xor or cmp would change ZF and could make you pick the wrong instruction.

Your turn, block B: load the guest’s registers

SDM: 27.4.1 (the registers the CPU does not load); 29.3 “Checking and Loading Guest State”.

Goal. Load the 15 general-purpose registers from *regs, so the guest sees the values runvm and the previous exits left there. (The CPU loads RSP, RIP and RFLAGS itself.)

What to do. Fifteen movq instructions, from memory at an offset from %rdi into the register, using the offset table in Section 2. One line, to show the syntax: movq 8(%rdi), %rbx loads the 8 bytes at address %rdi + 8 into %rbx. In struct guestregs, offset 8 is rbx, so this loads the guest’s rbx.

The ordering hazard. %rdi is the pointer you use to address *regs. Loading the guest’s %rdi overwrites that pointer. If you load it early, every later movq N(%rdi), ... reads from the guest’s rdi value instead, and loads garbage (or faults). So the load of %rdi must come last. Every other register can go in any order. After the last movq, the register that held the pointer holds the guest’s value, and you cannot use it for anything else.

Hints.

• Take the offsets from the table in Section 2, not from memory. Remember that rdi is at 40 and rbp is at 48.
• Because %rsi is overwritten here, this is the moment the earlier testl mattered.
• Do not load %rsp from *regs. There is no such field. The CPU gets the guest’s stack pointer from the VMCS.
• Do not add a push or pop here. The stack must stay exactly as your block A left it.

Given: launch and failure path

  # Enter the guest. ZF still holds the result of testl above.
  jnz   1f                    # launched != 0: already launched, resume
  # VMLAUNCH is required for the first entry after VMCLEAR (the VMCS
  # launch state is "clear"); VMRESUME is required afterwards. The
  # wrong one fails with a VM-instruction error. Because vmrun
  # VMCLEARs the VMCS after every exit, it always passes launched = 0.
  # On success neither instruction falls through: the guest starts
  # running and control returns only via a VM exit to vmx_exit.
  vmlaunch
  jmp   2f                    # reached only if VMLAUNCH failed
1:
  vmresume                    # reached only if launched != 0
2:
  # VMLAUNCH/VMRESUME fell through: the entry failed (CF or ZF set).
  # CF = 1 is "VMfailInvalid" (no current VMCS), ZF = 1 is
  # "VMfailValid" (error number in VM_INSTRUCTION_ERROR). We do not
  # look at the flags: reaching this line at all means failure, and
  # the C caller reads VM_INSTRUCTION_ERROR. The guest has not run, so
  # *regs was not modified. Unwind exactly what we pushed:
  addq  $8, %rsp              # drop the saved regs pointer
  popq  %r15                  # restore host callee-saved registers,
  popq  %r14                  # in reverse order of the pushes
  popq  %r13
  popq  %r12
  popq  %rbx
  popq  %rbp
  movl  $-1, %eax             # return value: -1 means entry failed
  ret                         # back to the C caller, on its stack

The jnz picks VMRESUME if launched was nonzero. Otherwise execution continues into VMLAUNCH. On success, neither instruction falls through: the CPU is now in the guest. If VMLAUNCH falls through, the entry failed, and the code after label 2 unwinds the frame that block A built and returns -1. The guest never ran, so *regs is unchanged.

This unwinding is your first test of block A. It pops exactly seven things in a particular order: it drops the regs slot with an addq, then pops %r15, %r14, %r13, %r12, %rbx, %rbp. If your pushes were in a different order, or you pushed a different number of times, vmx_enter returns with a wrong %rsp or with a corrupted callee-saved register, even when the entry fails cleanly.

Your turn, vmx_exit: save the guest and return 0

SDM: 30.1 and 30.3 (what an exit saves); 30.5.3 (what an exit loads into RSP and RFLAGS).

Goal. Store the guest’s 15 general-purpose registers into *regs, restore the host’s callee-saved registers, and return 0 from vmx_enter.

State on entry to vmx_exit. The CPU has loaded %rsp = HOST_RSP, which is the frame from block A. So (%rsp) holds the regs pointer. Every general-purpose register still holds a guest value, including %rdi. There is no free register.

What to do, in order:

1. Get the regs pointer into a register without losing any guest value.
2. Store the other 14 registers at their offsets.
3. Store the guest’s rdi at offset 40.
4. Discard the regs slot, pop the six callee-saved registers, set the return value to 0 and ret.

Hints.

• The problem in step 1. To store anything, you need a register that holds the address of regs. But all 15 registers hold guest values that must be saved first. The trick: free one register by pushing it on the stack. Push %rdi. That saves the guest’s rdi in memory (it is a push onto the host stack, below your frame), and %rdi is now free to hold the pointer.
• After that push, the stack has two things on top: the saved guest rdi at 0(%rsp) and regs at 8(%rsp). Load regs into %rdi from 8(%rsp). Draw the stack on paper before you write this.
• Now store the 14 other registers, using the offsets from Section 2. One line, to show the syntax: movq %rbx, 8(%rdi) stores %rbx into the 8 bytes at address %rdi + 8, which is the rbx field of *regs. For rdi, you pop the saved value back into some register that you have already saved (the order matters: which register is it safe to overwrite?) and store it at offset 40.
• When you restore the host registers, the stack must be exactly as it was right after block A’s last push. Count your pushes and pops. If you left the temporary rdi slot on the stack, the unwinding is off by one slot, and every restored register is wrong.
• The unwinding is the same as in the failure path: addq over the regs slot, then the six pops in reverse order. The only difference is the return value: 0, not -1. xorl %eax, %eax does it.
• The guest’s rax, rbx, … are stored from the registers where they were left when the exit happened. Do not run any code that uses a register before it is stored, except for the single register freed by the push.
• At vmx_exit, %rax to %r15 hold guest values that C code knows nothing about. Do not call C functions from here.

Checkpoint

Build, boot, and run runvm helloguest. (If you use Bochs, remember it is slow; see the “Where to run it” section.)

The first thing the guest does is read IA32_EFER with RDMSR at guest address 0x100016. We configured the VMCS so that RDMSR always exits (there is no MSR bitmap), so this is the first VM exit. It is basic exit reason 31 (EXIT_RDMSR, “RDMSR” in the SDM’s table of basic exit reasons, Appendix C).

What you see depends on how much of the rest of the system you have built:

• If vmxhandle() is not written yet (it only returns 0), vmrun() returns the exit to runvm, which prints:

  runvm: unhandled exit 31, qualification 0x0, rip 0x100016
  

  This is the success message for this part. It means that the guest ran, the exit came back to vmx_exit, the stack unwound correctly, and vmrun() read a sensible exit reason and guest RIP. A qualification of 0 is correct for RDMSR.

• If vmxhandle() is already in place, the exit is emulated and the guest goes on. You see Hello from a VT-x guest and runvm: guest halted at rip 0x10004E.

Either way, the rest of xv6 must still work after runvm exits: the shell prompt comes back, and you can run runvm helloguest a second time. If the second run misbehaves, the host registers were not restored properly.

Two experiments are worth doing:

1. In vmxhoststate(), temporarily write HOST_CS_SELECTOR as (SEG_KCODE << 3) | 3 and run again. Explain the message you see. (Hint: 29.2.3, and error numbers in 33.4.)
2. Add a cprintf in vmrun() after vmx_enter returns that prints the guest’s regs.rcx. What value do you expect for the first exit, and why?

If something fails:

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. struct guestregs has no rsp field. Why can the guest’s stack pointer not be handled the way the other registers are? Where does the CPU keep it instead, and what would go wrong if vmx_exit tried to store %rsp into *regs like the others?
2. HOST_RIP is written once, in vmcsinit(), but HOST_RSP is written in vmx_enter on every entry. What makes one constant and the other not? Describe a concrete situation in which the HOST_RSP written by a previous entry would be wrong.
3. vmrun() always calls vmx_enter(&vm->regs, 0), and so always uses VMLAUNCH. Which instruction clears the launch state, and where does vmrun() execute it? What would the VM-instruction error be if the code passed launched = 1 on the first entry?
4. vmxhoststate() is called before every entry, not once. Name two fields it writes that can differ between two calls to vmrun(), and say what is different (CPU, process, or both) for each. What would happen at the next VM exit if the field kept the old value?

Part 5: Running the guest, the vmrun loop and exit handling

In this part the guest runs. The loop that enters the guest, waits for a VM exit, saves what the guest left behind, and decides whether the kernel can deal with the exit itself or must hand it to runvm, is vmrun(). You have been using it since part 4, because it is the only way to test the assembly. It is given to you complete, with one planted bug that you find and fix here. You write the pieces it calls: the register accessors and the exit handlers. When you finish, the kernel can run helloguest from its first instruction to its final HLT.

By now you have VMX turned on (part 1), a VMCS page (part 2), the control fields and guest state (part 3), and guest memory behind an EPT (part 4). What is missing is the code that uses them all, over and over.

What is given and what you write

All of this is in vmx.c. We give you the entry/exit assembly (vmx_enter and vmx_exit in vmxasm.S), the VMX instruction wrappers (vmptrld(), vmclear(), vmread(), vmwrite(), invept()), vmxhoststate(), getvm(), iomask(), and the system-call glue in sysvmx.c. vmrun() is also given (with the planted bug). You write four functions. In the handout’s source tree each has its body stripped to a stub that does the safest possible thing:

Two spots are marked Your turn: the planted bug in vmrun() (Step 8) and the I/O case of vmxhandle(). For those we give the goal, the manual sections and hints, but no code. The other blocks in vmxhandle(), vmxmsr(), vmsetreg() and vmgetreg() are to be pasted, in order, into the function they are listed under. The steps of vmrun() are a walk-through of code that is already in your tree: read them.

Where to look things up. As before, “SDM” is the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026). Section numbers differ between editions, so if you use another one, search for the titles.

Section A: The register cache in struct vm

Before the loop, look at struct vm near the top of vmx.c. The part that matters here is:

  /*
   * Guest state kept here between runs, and copied to and from the
   * VMCS around each entry.  The general-purpose registers are not in
   * the VMCS at all: the processor leaves them alone across entry and
   * exit, so vmxasm.S loads and saves them by hand from/to regs.
   * RIP, RSP, RFLAGS and EFER do live in the VMCS (guest-state area).
   */
  struct guestregs regs;
  uint64 rip, rsp, rflags, efer;

This is the register cache: a copy of the guest’s registers kept in ordinary kernel memory. Why have one? Two reasons.

First, the general-purpose registers (RAX to R15, except RSP) are not in the VMCS at all. The processor leaves them alone on entry and exit, so vmx_enter and vmx_exit load and save them by hand, from and to vm->regs. That is the only place they live between runs.

Second, RIP, RSP, RFLAGS and EFER are in the VMCS, but the VMCS is not readable between runs. A VMCS is only the current VMCS on a CPU between VMPTRLD and VMCLEAR, and VMREAD and VMWRITE work only on the current one. Our loop clears the VMCS after every exit (you will see why below), so between vmrun() calls no VMCS is current on any CPU. The manual also says software must never read VMCS data with ordinary memory accesses, because the format is implementation specific and the processor may keep some of the data on the chip. So when runvm wants to set RIP before the first run, or read RAX after an IN exit, the kernel has nowhere to put or find the value except struct vm.

The rule is simple. vmrun() copies RIP, RSP, RFLAGS and EFER into the VMCS just before each entry, and copies them back into struct vm right after each exit. The two system calls vmsetreg and vmgetreg read and write the cache.

SDM: 27.11.1 “Software Use of Virtual-Machine Control Structures”; 27.1 “Overview” (the VMCS is “active” and “current” only between VMPTRLD and VMCLEAR).

vmsetreg()

  if((vm = getvm(id)) == 0 || reg < 0 || reg >= VMREG_NREGS)
    return -1;
  switch(reg){
  case VMREG_RSP:
    vm->rsp = val;
    break;
  case VMREG_RIP:
    vm->rip = val;
    break;
  case VMREG_RFLAGS:
    /* RFLAGS bit 1 always reads as 1; VM entry checks for it. */
    vm->rflags = val | 0x2;
    break;
  default:
    /* VMREG_RAX..VMREG_R15 follow the layout of struct guestregs. */
    ((uint64*)&vm->regs)[reg] = val;
  }
  return 0;

getvm() checks that the VM exists and belongs to the calling process. We also range-check reg before using it, because it comes straight from user space.

The three special registers each get a case, since they are separate fields. For everything else, the numbering of VMREG_RAX to VMREG_R15 in vmxapi.h was chosen to equal the order of the fields of struct guestregs. The register number is therefore an index into that structure treated as an array of uint64. (Note that this is not the hardware register numbering: RBX is second and RDI comes before RBP. If you reorder either list, vmx_enter and vmx_exit will silently load the wrong registers.)

One detail: RFLAGS bit 1 is reserved and always 1. VM entry checks that the guest RFLAGS field has it set, so we force it on. A guest that starts with RFLAGS = 0 would otherwise fail to enter.

vmgetreg()

  if((vm = getvm(id)) == 0 || reg < 0 || reg >= VMREG_NREGS)
    return -1;
  switch(reg){
  case VMREG_RSP:
    *val = vm->rsp;
    break;
  case VMREG_RIP:
    *val = vm->rip;
    break;
  case VMREG_RFLAGS:
    *val = vm->rflags;
    break;
  default:
    /* VMREG_RAX..VMREG_R15 follow the layout of struct guestregs. */
    *val = ((uint64*)&vm->regs)[reg];
  }
  return 0;

This is the mirror image. The values are the ones saved at the last exit, which is exactly what a monitor needs to complete an IN: read RAX, merge in the new bytes, write RAX back (completein() in runvm.c does that). Notice that sys_vmgetreg (given) is responsible for copying *val out to user space; this function only fills a kernel pointer.

Section B: The vmrun() loop

vmrun() is already in your tree. Read the steps below against it, and find the planted bug in Step 8. vmrun() is the heart of the hypervisor. It is called once per system call, and it loops internally. Each pass of the loop is one round trip: VM entry, guest runs, VM exit. The loop stays inside the kernel for the exits it can handle (an interrupt, an EFER access) and returns to user space for the rest.

The function starts with locals that we declare for you:

  struct vm *vm;
  struct vmexit e;
  uint64 reason, entryctl;
  uint inslen;
  int c, r;

Here e collects everything we learn about this exit, inslen is the length of the instruction that caused it, c is the CPU number, and r holds the result of vmx_enter.

Three invariants hold on every pass. Keep them in mind, because each step below exists to keep one of them true:

1. Interrupts are off while the VMCS is current, so the process cannot be moved to another CPU in the middle.
2. The host state is rewritten before every entry, because it depends on which CPU and which process we are on.
3. The VMCS is cleared after every exit, before interrupts are turned back on, so that no CPU keeps a cached copy of it.

Because of the third rule, the VMCS is always in the “clear” launch state when we enter the guest, so we always use VMLAUNCH and never VMRESUME. (vmx_enter takes the choice as its second argument; we always pass 0.)

Step 1: Find the VM and start the loop

SDM: 27.11.3 “Initializing a VMCS” is background only; this step is xv6 code (the killed flag in proc.h).

  if((vm = getvm(id)) == 0)
    return -1;

  for(;;){
    /* Check for kill between guest runs, since the guest may spin. */
    if(myproc()->killed)
      return -1;
    memset(&e, 0, sizeof(e));
    inslen = 0;

getvm() rejects a bad id or a VM that belongs to another process. In the loop, the first thing we do is check killed. A guest may run forever (an endless loop with no exits), and the only way to stop it is for the host timer interrupt to cause an exit, after which we come back here. If the process was killed, we leave. Then we clear e, so that no field from a previous pass leaks into this exit’s report.

Step 2: Disable interrupts and load the VMCS

SDM: 27.1 “Overview” (current VMCS, VMPTRLD); 33.3 “VMX Instructions”, VMPTRLD.

    /*
     * Disable interrupts: from VMPTRLD until VMCLEAR the VMCS is
     * bound to this CPU, so we must not be rescheduled elsewhere.
     */
    pushcli();
    c = cpuid();
    if(!cpus[c].vmx_ok || vmptrld(V2P(vm->vmcs)) < 0){
      popcli();
      return -1;
    }

pushcli() turns interrupts off (xv6 counts nested calls; a matching popcli() turns them back on only if they were on before). From now on the process cannot be rescheduled, so it stays on one CPU. We then read the CPU number after pushcli(), so it cannot change under us.

We check that this CPU did enter VMX operation (part 1 set vmx_ok), and VMPTRLD makes our VMCS the current VMCS of this CPU. The wrapper takes the VMCS’s physical address, hence V2P(). If either check fails, we undo pushcli() before returning. Forgetting that popcli() on an error path leaves interrupts off on this CPU forever.

Step 3: Flush stale EPT translations, once per CPU

SDM: 31.4.3.4 “Guidelines for Use of the INVEPT Instruction”; 33.3 “VMX Instructions”, INVEPT.

    /*
     * Guest-physical translations are cached per CPU and tagged by
     * EPTP.  Flush any left from an earlier VM whose EPT used the same
     * page, the first time this VM runs on this CPU.
     * (SDM 31.4.3.4: software must INVEPT when it changes mappings or
     * reuses an EPT root.)  One bit per CPU records that we did so.
     */
    if(!(vm->flushed & (1 << c))){
      invept(INVEPT_SINGLE_CONTEXT, vm->eptp);
      vm->flushed |= 1 << c;
    }

A CPU caches guest-physical to host-physical translations, tagged with the EPT pointer. If an earlier VM (now freed) used the same page as its EPT root, a CPU might still hold translations that belong to that old VM. They would be wrong for ours. INVEPT with the “single-context” type discards the cached entries for one EPT pointer.

We need this only the first time this VM runs on a given CPU, so the field vm->flushed holds one bit per CPU. Each CPU does the flush once, then remembers it did. (The same instruction would be required if you changed mappings of a running VM; this homework does not.)

Step 4: Write the host state and copy guest state into the VMCS

SDM: 27.5 “Host-State Area”; 27.4 “Guest-State Area”; 30.5 “Loading Host State”.

    /* Host state: CPU- and process-specific, so rewrite it every time. */
    vmxhoststate();
    /* Guest state that changes outside the guest: copy into the VMCS. */
    vmwrite(GUEST_RIP, vm->rip);
    vmwrite(GUEST_RSP, vm->rsp);
    vmwrite(GUEST_RFLAGS, vm->rflags);
    vmwrite(GUEST_IA32_EFER, vm->efer);

vmxhoststate() (given) writes what the processor loads on a VM exit: the kernel’s CR0, CR3 and CR4, the segment selectors, and the bases of TR, GDTR, IDTR and the FS and GS MSRs. Most of these differ per CPU (every CPU has its own TSS, GDT and per-CPU GS base) and CR3 differs per process. Because we VMCLEAR after every exit, the process may come back on a different CPU next time, so we cannot write these once. Writing them every time is cheap and removes a whole class of bugs.

Then we move the four cached values into the guest-state area, which is what the guest will see when it starts running. This is the “copy in” half of the register cache.

Step 5: Set the entry control “IA-32e mode guest” from EFER.LMA

SDM: 27.8 “VM-Entry Control Fields”; 29.3.1.1 “Checks on Guest Control Registers, Debug Registers, and MSRs” (the EFER check).

    /*
     * "IA-32e mode guest" must match EFER.LMA, which the guest sets
     * by enabling paging with EFER.LME = 1.  (VM-entry checks, SDM
     * 29.3.1.1.)  So recompute the control from the saved EFER on
     * every entry: clear the bit, then set it if LMA is on.
     */
    entryctl = vmread(VM_ENTRY_CONTROLS) & ~(uint64)ENTRY_IA32E_MODE_GUEST;
    if(vm->efer & EFER_LMA)
      entryctl |= ENTRY_IA32E_MODE_GUEST;
    vmwrite(VM_ENTRY_CONTROLS, entryctl);

When VM entry loads the guest’s IA32_EFER, the processor checks that EFER.LMA (bit 10) equals the VM-entry control “IA-32e mode guest”. If they disagree, the entry fails with “invalid guest state”. The guest sets LMA itself, by turning on paging while LME is set (see helloguestasm.S), so we cannot know it in advance. Instead we recompute the control on every entry: read the current value, clear the bit, set it if the cached EFER has LMA, and write it back.

Without this step, the first entry works (LMA is 0, the control is 0). The guest then enables long mode, so the saved EFER has LMA set, and the next entry (after the guest’s next exit) fails with an entry-failure exit, reason 33. If you see that, this step is the place to look.

Step 6: Enter the guest

SDM: 29.1 “Basic VM-Entry Checks” (VMLAUNCH needs a VMCS in the “clear” launch state); 27.1 “Overview” (launch state); 33.3, VMLAUNCH.

    /*
     * Enter the guest (vmxasm.S).  The second argument is "launched":
     * 0 means VMLAUNCH.  Because we VMCLEAR after every exit, the VMCS
     * is always in the clear state here, so we never use VMRESUME.
     * Returns 0 after a VM exit, or -1 if the entry instruction itself
     * failed (invalid controls or host/guest state).
     */
    r = vmx_enter(&vm->regs, 0);

One line. vmx_enter (given, vmxasm.S) saves the host’s callee-saved registers, tells the VMCS where to land on the exit (HOST_RSP), loads the guest’s registers from vm->regs, and executes VMLAUNCH. The call does not return until the guest exits. It then returns 0, with the guest’s registers saved back into vm->regs. If VMLAUNCH itself fails, it returns -1 immediately and the guest never ran.

We always pass launched = 0, because we VMCLEAR after every exit.

Step 7: Read the exit information and save the guest state

SDM: 27.9.1 “Basic VM-Exit Information”; 30.2.1 “Basic VM-Exit Information”; 27.9.4 “Information for VM Exits Due to Instruction Execution”; 27.9.5 “VM-Instruction Error Field”; 33.4 “VM Instruction Error Numbers”; 29.8 “VM-Entry Failures During or After Loading Guest State”.

    /*
     * Back in the host.  The VMCS is still current on this CPU, so
     * read everything we need now, before VMCLEAR.
     */
    if(r < 0){
      /* VMLAUNCH failed: report the VM-instruction error (SDM 33.4). */
      e.reason = EXIT_VMFAIL;
      e.error = vmread(VM_INSTRUCTION_ERROR);
      e.rip = vm->rip;
    } else {
      /*
       * Exit reason: bits 15:0 are the basic reason, and bit 31 is
       * set if this was a failed VM entry (SDM 30.2.1), for example
       * invalid guest state.
       */
      reason = vmread(VM_EXIT_REASON);
      e.reason = reason & 0xFFFF;
      e.entryfail = (reason & EXIT_REASON_ENTRY_FAILURE) != 0;
      /* Extra detail whose meaning depends on the exit reason. */
      e.qual = vmread(EXIT_QUALIFICATION);
      e.intrinfo = vmread(VM_EXIT_INTR_INFO);
      e.intrerr = vmread(VM_EXIT_INTR_ERROR_CODE);
      /* Faulting guest-physical address, for EPT violations. */
      e.gpa = vmread(GUEST_PHYSICAL_ADDRESS);
      /* Length of the exiting instruction, to skip it. */
      inslen = vmread(VM_EXIT_INSTRUCTION_LEN);
      /* Save the guest state the processor stored in the VMCS. */
      vm->rip = vmread(GUEST_RIP);
      vm->rsp = vmread(GUEST_RSP);
      vm->rflags = vmread(GUEST_RFLAGS);
      /* Saved because of EXIT_SAVE_IA32_EFER; picks up a new LMA. */
      vm->efer = vmread(GUEST_IA32_EFER);
      e.rip = vm->rip;
    }

Two things can have happened.

r < 0: the VMLAUNCH instruction failed. The processor rejected the VMCS (bad controls or bad host state) before the guest executed anything. The VM-instruction error field says why (SDM 33.4 lists the numbers; 4 is “VMLAUNCH with non-clear VMCS”, 7 is “VM entry with invalid control field(s)” and 8 is “VM entry with invalid host-state field(s)”). We mark this with our own pseudo exit reason EXIT_VMFAIL, which is 0xFFFF and so cannot be confused with a real 16-bit reason.

Otherwise, a real VM exit. The exit reason field has the basic reason in bits 15:0. Bit 31 is set when the exit is really a failed entry (for example invalid guest state, reason 33), which we record in e.entryfail. The other fields give detail whose meaning depends on the reason: the exit qualification (the I/O port and size, for example), the interruption information (the vector of a guest exception), the faulting guest-physical address (EPT violations) and the instruction length. Then we copy RIP, RSP, RFLAGS and EFER out of the VMCS into the cache. This is the “copy back” half. The EFER value includes whatever LMA the guest just set, because we asked for the guest’s EFER to be saved on exit (EXIT_SAVE_IA32_EFER in part 3).

Do all of this now. The VMCS is still current on this CPU and the reads are legal. After VMCLEAR they are not.

Step 8 (Your turn): Find the planted bug

SDM: 27.11.1 “Software Use of Virtual-Machine Control Structures”; 27.1 “Overview”; 33.3 “VMX Instructions”, VMCLEAR.

This is where the loop’s most important invariant is enforced, and it is two lines. In your tree they are in the wrong order: that is the planted bug. The order is the whole point.

Goal. After the information is read, vmrun() gives up the VMCS: it makes sure that no CPU keeps it cached (vmclear()), and lets the CPU take interrupts again (popcli()). Find these two lines in vmrun(), decide which order is right, and fix it. The VMCS physical address is V2P(vm->vmcs), as in Step 2.

The design question. Which of the two goes first, and what goes wrong with the other order? Think it through before you change anything:

• After popcli(), interrupts are on again (if they were on when pushcli() ran), and xv6’s timer interrupt can end up in yield(). Where can the process be running the next time it is scheduled?
• Where was our VMCS made current, and what does the manual say a logical processor may keep for a VMCS that is active on it? Read the first paragraph of SDM 27.11.1. What does the manual require before another CPU executes VMPTRLD on the same VMCS, and what does it say may happen if that is not done?
• What does VMCLEAR do to the launch state, and why does that make the 0 we pass to vmx_enter correct?

Hints.

• One of the two orders is correct on every path. The other works for months on a one-CPU machine and fails randomly on four.
• The two calls must be on every path out of the entry, including the case where VMLAUNCH failed. In our code both paths reach this point, so one pair of calls is enough.
• The comment that the next block carries (“a pending host interrupt is delivered here”) tells you something about what popcli() does.

Checkpoint for this step. With one CPU (CPUS=1) both orders may appear to work. Boot with CPUS=4 and run runvm helloguest several times; the wrong order may eventually show up as a VM-entry failure, an odd exit reason, or a hang (the manual calls the result undefined, so the symptom can vary). Be ready to explain why it is wrong even on the runs that happen to work.

Step 9: Decide: handle the exit or return it

SDM: 30.2.1 “Basic VM-Exit Information”; Appendix C “Basic Exit Reasons”.

    /*
     * Failed entries and VMLAUNCH failures always go to user space.
     * Otherwise vmxhandle() decides; returning 1 loops to re-enter.
     */
    if(e.reason == EXIT_VMFAIL || e.entryfail || !vmxhandle(vm, &e, inslen))
      break;
  }
  /* Report the exit that ended the loop to the user program. */
  *ue = e;
  return 0;

A failed entry or failed VMLAUNCH is never something the kernel can repair, so those always break out of the loop. For a normal exit, vmxhandle() returns 1 to say “I dealt with it, run the guest again” or 0 to say “give this to user space”. After the loop we copy e out, and return 0. (sys_vmrun, given, copies it to the user’s struct vmexit.)

Note that vmxhandle() is called after the VMCS has been cleared and interrupts are enabled again. It must therefore only use e, inslen and the cache in struct vm, never vmread. This is also why Step 7 reads inslen before the VMCS is cleared.

Section C: Handling exits

vmxmsr(): emulating EFER

SDM: 28.1.3 “Instructions That Cause VM Exits Conditionally” (RDMSR, WRMSR); 29.3.1.1 (EFER checks); Appendix C, reasons 31 and 32.

  uint64 val;

  if((uint)vm->regs.rcx != EFER_MSR)
    return -1;
  if(write){
    val = (vm->regs.rdx << 32) | (uint)vm->regs.rax;
    /*
     * The processor sets LMA itself when the guest enables paging
     * with LME set; LMA is read-only to software, so preserve ours.
     */
    vm->efer = (val & ~(uint64)EFER_LMA) | (vm->efer & EFER_LMA);
  } else {
    /* Return the saved value; writes to EAX/EDX zero the upper halves. */
    vm->regs.rax = (uint)vm->efer;
    vm->regs.rdx = vm->efer >> 32;
  }
  return 0;

We did not set up an MSR bitmap, so every RDMSR and WRMSR in the guest causes a VM exit. The guest’s trampoline uses exactly two: it reads IA32_EFER, sets LME, and writes it back. We emulate only EFER and send any other MSR to user space (return -1).

Both instructions take the MSR number in ECX and the 64-bit value in EDX:EAX. The handler reads ECX from the register cache, since the guest’s registers were saved by vmx_exit. A WRMSR builds the 64-bit value from EDX:EAX, and stores it in vm->efer; the next vmrun() step 4 copies it into the VMCS. An RDMSR returns the cached EFER by writing it into the cached RAX and RDX. The (uint) casts matter: they keep only the low 32 bits of the registers, as the instruction defines.

Why preserve LMA? EFER.LMA is “long mode active”. Software cannot set it: the processor does, when paging is enabled with LME set. In helloguest the RDMSR/WRMSR pair runs before paging is turned on, so our cached LMA is still 0 and the mask changes nothing. It matters for a guest that writes EFER again once long mode is active: a plain copy would let the written LMA bit replace the one the processor set, and the cached EFER could then disagree with the entry control “IA-32e mode guest” (Step 5), which makes the next entry fail the EFER check of SDM 29.3.1.1. Masking out the guest’s LMA bit and keeping ours means that LMA always reflects what the processor last told us.

vmxhandle(): the dispatch

SDM: Appendix C “Basic Exit Reasons”; 30.2.5 “Information for VM Exits Due to Instruction Execution”; 30.3.3 “Saving RIP, RSP, RFLAGS, and SSP”.

vmxhandle() is one switch on the basic exit reason. It returns 1 to resume the guest and 0 to return to user space. The opening and the default:

  switch(e->reason){
  ...
  default:
    /* Anything else (EPT violation, CPUID, triple fault, ...). */
    return 0;
  }

Every case below goes inside the switch, before default.

Why RIP must be advanced. When an exit is caused by executing an instruction (HLT, IN, OUT, RDMSR, WRMSR, …), the RIP the processor saves references that instruction (SDM 30.3.3). The instruction has not completed. If we resumed the guest with that RIP, it would execute the same instruction again, exit again, and loop forever. So, after emulating the instruction, we add the instruction length (the VM_EXIT_INSTRUCTION_LEN field we read in Step 7, passed in as inslen) to the cached RIP. The field is meaningful only for exits that come from executing an instruction (SDM 30.2.5), which is why Step 1 sets inslen to 0 at the start of each pass.

External interrupt and NMI

  case EXIT_EXTERNAL_INTR:
    /*
     * The interrupt is still pending in the local APIC (we don't
     * acknowledge it on exit), so the host took it at popcli().
     * Nothing to do but resume the guest.
     */
    return 1;
  case EXIT_EXCEPTION_NMI:
    /*
     * This reason covers both guest exceptions (which we send to user
     * space for debugging) and NMIs, told apart by the interruption
     * type in the exit interruption-information field.
     */
    if(INTR_INFO_TYPE(e->intrinfo) == INTR_TYPE_NMI)
      return 1;  /* a host NMI; xv6 has no use for it */
    return 0;

SDM: 28.2 “Other Causes of VM Exits” (external interrupts, NMIs, exceptions); 30.1 “Architectural State Before a VM Exit” (interrupt not acknowledged); 30.5.3 (RFLAGS cleared on exit); 27.9.2 “Information for VM Exits Due to Vectored Events” (the interruption-information field).

In part 3 we asked for exits on external interrupts, so the host kernel (not the guest) gets them. Think about what has to happen. The timer fires while the guest is running. The processor exits to us. Our host RFLAGS has interrupts off (the exit clears RFLAGS except bit 1, SDM 30.5.3). We did not set “acknowledge interrupt on exit”, and the manual says that without it the interrupt is not acknowledged and remains pending (SDM 30.1). So nothing needs to be done with it here. Back in vmrun(), the popcli() of Step 8 turns interrupts on and the pending interrupt is delivered through the normal kernel IDT, with the normal xv6 handler, which may even reschedule us. Then vmxhandle() returns 1, and the loop enters the guest again.

This is also why the interrupt exit costs us nothing special: it is only a trip out and back in.

An NMI arrives as the “exception or NMI” reason, shared with real guest exceptions. The interruption type (bits 10:8 of the interruption information) tells them apart. An NMI is a host event, so we ignore it and resume. A real guest exception goes to user space: our guest has no IDT, so any exception is a bug that runvm should report.

RDMSR and WRMSR

  case EXIT_RDMSR:
  case EXIT_WRMSR:
    /*
     * Emulate, then advance RIP past the RDMSR/WRMSR (the processor
     * does not, since it was an exit rather than a completed
     * instruction).  Unsupported MSRs go to user space.
     */
    if(vmxmsr(vm, e->reason == EXIT_WRMSR) < 0)
      return 0;
    vm->rip += inslen;
    return 1;

We call vmxmsr(). If it cannot handle the MSR, the exit goes to user space with RIP unchanged. If it can, we skip the instruction and resume without returning to user space; the guest does not notice.

HLT

  case EXIT_HLT:
    /* Skip the HLT so a later vmrun() continues after it. */
    vm->rip += inslen;
    return 0;

SDM: 28.1.3 (HLT exiting).

We asked for exits on HLT in part 3, so the guest can never actually halt the CPU. We skip it so that if runvm calls vmrun() again, the guest continues after the HLT. runvm treats this exit as the end of the program.

I/O (Your turn)

SDM: 28.1.3 (I/O instructions; “unconditional I/O exiting”); Table 30-5 “Exit Qualification for I/O Instructions” in 30.2.1; Appendix C, reason 30; 30.2.5 for the instruction length.

This is the last case. Its context is:

  case EXIT_IO:
    /* ... your code ... */
    return 0;

Goal. The guest executed IN or OUT. Fill in e->io (struct vmexit in vmxapi.h: port, size, in, data) so runvm can emulate the device, advance the guest past the instruction, and send the exit to user space.

What to do, in order.

1. Read Table 30-5. The exit qualification, which is already in e->qual, packs the access size, the direction, a string-instruction flag and the port. Work out which bits give each field, and note the encoding of the size field (it is not the size in bytes).
2. String instructions (INS, OUTS) move data between memory and the port. They need extra information (a memory address, a repeat count) that we do not decode. What should the kernel do when the string bit is set? Which return value, and does RIP move?
3. Decode port, size and in. Use the table: one of them needs an adjustment, one needs a shift, one a mask.
4. Advance the guest’s RIP past the instruction. Which value tells you how far? Look at what the other cases do.
5. Decide what the kernel has to record, depending on direction:
   • For an OUT, the guest has sent a value that runvm needs to see (the byte for the serial port). Where is that value, according to the OUT instruction? Only the low size bytes are the data; iomask(size) (given) gives the mask. Store it in e->io.data.
   • For an IN, the kernel records nothing. The value belongs to a device that lives in user space. runvm will compute it and write it into the guest’s RAX with vmsetreg() before the next vmrun() (runvm does that, as you will read in part 6; it is also why vmgetreg exists).
6. Return 0, so the exit reaches user space.

Hints.

• In the table the direction bit is 0 for OUT and 1 for IN. Check which value e->io.in should take.
• If you forget to advance RIP, the guest re-executes the same OUT and runvm prints the first character forever.
• Advancing by a wrong fixed amount makes the guest resume in the middle of an instruction. OUT with DX is one byte, but OUT with an immediate port is two.
• e->io.data is only meaningful for OUT. Do not write it for IN.
• Do not call vmread() here. The VMCS is no longer current.

Checkpoint for this case. Given the exit trace in the Checkpoint section below, the 24 I/O exits must carry the characters of Hello from a VT-x guest one at a time. Print e->io.data in a temporary cprintf to confirm.

Checkpoint

runvm is given to you complete (you read it in part 6), so as soon as the kernel half works, runvm helloguest prints the greeting. Before you rely on that, use a temporary trace to check the kernel by itself.

At the top of vmxhandle(), before the switch, add:

  cprintf("exit %d rip %x\n", (uint64)e->reason, e->rip);

(%x reads 64 bits, so cast.) Rebuild, boot with CPUS=1 if you want a clean trace, and run runvm helloguest. You should see, in this order:

1. exit 31 (RDMSR of EFER), then exit 32 (WRMSR of EFER). Both are handled in the kernel and runvm never sees them.
2. Then 24 lines of exit 30 (I/O). One OUT per character of Hello from a VT-x guest\n. The RIP changes between lines only slightly, since the loop is tight.
3. Last, exit 12 (HLT). runvm then prints runvm: guest halted at rip 0x... and exits.

The host timer can also cause an exit 1 (external interrupt) at any point. A few of those, especially in Bochs, are normal and harmless: they mean Step 8 and the external-interrupt case work. If the trace never shows exit 1 over a long run, check that PIN_EXTINT_EXITING is set (part 3).

Remove the cprintf and run runvm helloguest again: it now prints Hello from a VT-x guest.

If something fails:

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. In Step 8, vmclear() must come before popcli(). Describe a concrete sequence of events, with two CPUs, in which the opposite order corrupts the VMCS. Which sentence of SDM 27.11.1 do you rely on?
2. We pass launched = 0 to vmx_enter on every entry, so we always use VMLAUNCH. What would the program have to do differently if we kept the VMCS loaded between exits and used VMRESUME instead? What would we give up?
3. On an external-interrupt exit, vmxhandle() does nothing but return 1, yet the interrupt is handled by the host before the guest runs again. Where and when exactly does the host handle it? What would change if the exit control “acknowledge interrupt on exit” were set?
4. The WRMSR exit and the OUT exit are both followed by vm->rip += inslen. What would happen if the code used a fixed length (for instance 2) instead? Name one instruction in helloguestasm.S or helloguest.c that shows the problem.

Part 6: The other half: the monitor (runvm) and the guest

Parts 1 to 5 built the kernel half of the hypervisor: VMX operation, a VMCS, EPT, and the run loop that enters the guest and handles the exits only the kernel can handle. In this part you build the other half, the part that makes it a usable system:

• runvm, a user program that plays the role QEMU plays on top of KVM. It loads a guest image, starts it, and emulates the guest’s “hardware”.
• the guest, a tiny program that prints a greeting through a (virtual) serial port.

When you finish, the whole homework works end to end:

$ runvm helloguest
Hello from a VT-x guest
runvm: guest halted at rip 0x10004E

This part has a very small amount of code to write. Most of it is a tour: you read the guest’s 32-bit to 64-bit transition and the plumbing that connects runvm to the kernel, because you need to understand them to debug anything that goes wrong in the earlier parts.

What is given and what you write.

Where to look things up. Each step lists the sections of the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3 (order number 325384-092, June 2026) that describe the mechanism it uses. We write “SDM” for it. Steps about the guest also cite Volume 3A (system architecture, paging, processor initialization), and one hint cites Volume 1 (basic architecture), which is a separate book. Section numbers change between editions, so if you use another one, search for the titles.

The split of work

Before the code, the picture. runvm and the kernel share the work the way QEMU and KVM do:

Every time the guest does something the kernel will not handle itself, vmrun() returns to runvm with a struct vmexit (in vmxapi.h) that says what happened. runvm emulates the event, then calls vmrun() again. That is the entire protocol.

runvm.c is given to you complete, so that you can run runvm helloguest as soon as the kernel half works (it is how you test parts 1 to 5). The only function in it left for you to write is completein(). Steps 1 to 4 walk through main() and doio(): read the code, and make sure you understand it. You do not paste these blocks.

Step 1: Read the image

SDM: none; this is ordinary xv6 file I/O. (The image is the input to the “guest memory” of Part 4.)

This is the start of main() in runvm.c:

  struct vmexit e;   /* filled in by the kernel at every VM exit */
  int fd, n, len, vm, memmb;

  /* Command line: the image file, and optionally the size of guest
   * physical memory in megabytes. */
  if(argc < 2 || argc > 3){
    printf(2, "usage: runvm image [memory-MB]\n");
    exit();
  }
  memmb = argc == 3 ? atoi(argv[2]) : 4;

  /* Read the whole image into memory. The kernel never opens files on
   * behalf of the hypervisor, so the monitor does the file I/O. The loop
   * reads until end of file or until the buffer is full. */
  if((fd = open(argv[1], O_RDONLY)) < 0){
    printf(2, "runvm: cannot open %s\n", argv[1]);
    exit();
  }
  for(len = 0; len < MAXIMAGE && (n = read(fd, image + len, MAXIMAGE - len)) > 0; len += n)
    ;
  close(fd);
  /* A full buffer means the file may have been truncated; refuse. */
  if(len == MAXIMAGE){
    printf(2, "runvm: %s is larger than %d bytes\n", argv[1], MAXIMAGE);
    exit();
  }

The command line is runvm image [memory-MB]. Guest memory defaults to 4 MB, which is plenty for the hello guest (it needs a bit over 1 MB of address space because it is loaded at 1 MB).

The kernel never opens files for the hypervisor. runvm reads the whole image into the static buffer image (256 KB) with read(), then hands the buffer to vmcreate(). This is the same division as in KVM: the VMM does file I/O, the kernel only runs the guest. The loop keeps reading until end of file or until the buffer is full. If the buffer filled up, the file may have been cut off, so we refuse to run it rather than start half a program.

struct vmexit e is declared here, on the stack. The kernel overwrites all of it on every vmrun().

Step 2: Create the VM and set RIP

SDM: none for this step; the interesting work happens inside vmcreate() in the kernel, which was built in the earlier parts.

  /* Step 1: create the VM. The kernel allocates the VMCS and the EPT
   * (guest-physical to host-physical page tables), allocates and zeroes
   * memmb megabytes of guest memory, and copies the image to
   * guest-physical address VM_LOAD_ADDR. Returns a VM id, like a file
   * descriptor, used by the other calls. */
  if((vm = vmcreate(image, len, memmb * 1024 * 1024)) < 0){
    printf(2, "runvm: vmcreate failed\n");
    exit();
  }
  /* Step 2: set the guest's initial registers. All other registers
   * keep the defaults chosen by the kernel (flat 32-bit protected mode,
   * paging off; see vmcsinit() in vmx.c). Execution starts at the first
   * byte of the image, the "start" label of helloguestasm.S. */
  vmsetreg(vm, VMREG_RIP, VM_LOAD_ADDR);

vmcreate() allocates the VM, builds its VMCS and EPT, allocates and zeroes memmb megabytes of guest memory, and copies the image to guest-physical address VM_LOAD_ADDR (0x100000, defined in vmxapi.h). It returns a small integer, the VM id, which works like a file descriptor: you pass it to the other calls.

vmsetreg(vm, VMREG_RIP, VM_LOAD_ADDR) points the guest’s instruction pointer at the first byte of the image. All other registers keep the defaults chosen by vmcsinit() (flat 32-bit protected mode, paging off; Step 7 explains why). We do not check the return value of vmsetreg() because with a valid vm and VMREG_RIP it cannot fail.

Pitfall. The monitor does not read ELF headers or anything else from the image. The first byte of the file is the first instruction. This is why the guest is a flat binary (Step 9) and why its link address must equal VM_LOAD_ADDR (Step 8).

Step 3: The run loop

SDM: Appendix C “VMX Basic Exit Reasons” for the numbers behind the EXIT_* names; 29.8 “VM-Entry Failures During or After Loading Guest State” for entry failures.

  /* Step 3: the run loop. Each vmrun() executes the guest until a VM
   * exit that the kernel cannot handle itself. */
  for(;;){
    if(vmrun(vm, &e) < 0){
      printf(2, "runvm: vmrun failed\n");
      exit();
    }
    /* A failed VM *entry* (rather than an exit from the guest). The
     * reason field then is the failure code, e.g. 33 for invalid guest
     * state; qual says which check failed. Usually a bug in the
     * guest-state setup. */
    if(e.entryfail){
      printf(2, "runvm: VM entry failed, reason %d, qualification 0x%x\n",
             e.reason, e.qual);
      exit();
    }
    /* Dispatch on the basic exit reason (EXIT_* in vmxapi.h). */
    switch(e.reason){
    case EXIT_IO:
      /* The guest executed IN or OUT: emulate the device, then resume
       * the guest (the continue goes to the next vmrun()). */
      doio(vm, &e);
      continue;
    case EXIT_HLT:
      /* The guest executed HLT. Our guest does so when it is done, so
       * treat this as normal termination. e.rip is the address of the HLT
       * itself; the kernel has already advanced the guest's RIP past it. */
      printf(1, "runvm: guest halted at rip 0x%x\n", e.rip);
      exit();
    case EXIT_VMFAIL:
      /* Not a hardware exit: the VMLAUNCH instruction itself failed
       * (bad VMCS contents). error is the VM-instruction error number
       * from the Intel SDM. */
      printf(2, "runvm: VMLAUNCH failed, VM-instruction error %d\n", e.error);
      exit();
    case EXIT_EXCEPTION_NMI:
      /* The guest took a CPU exception. The hypervisor intercepts all
       * of them since the guest has no IDT. The low 8 bits of the
       * interruption information are the vector (14 = page fault...). */
      printf(2, "runvm: guest exception %d, error code 0x%x, rip 0x%x\n",
             e.intrinfo & 0xff, (uint64)e.intrerr, e.rip);
      exit();
    case EXIT_TRIPLE_FAULT:
      /* The guest faulted while delivering a fault; a real CPU would
       * reset. Common cause: running with a broken page table or GDT. */
      printf(2, "runvm: guest triple fault at rip 0x%x\n", e.rip);
      exit();
    case EXIT_EPT_VIOLATION:
      /* The guest touched guest-physical memory that has no EPT
       * mapping, i.e. beyond the memory size given to vmcreate(). gpa
       * is the offending guest-physical address. */
      printf(2, "runvm: EPT violation at guest-physical 0x%x, rip 0x%x\n",
             e.gpa, e.rip);
      exit();
    }
    /* Any other reason (CPUID, a WRMSR the kernel does not emulate,
     * ...) is not supported by this monitor: report it and give up. */
    printf(2, "runvm: unhandled exit %d, qualification 0x%x, rip 0x%x\n",
           e.reason, e.qual, e.rip);
    exit();
  }

Each trip around the loop runs the guest until something the kernel cannot handle itself. The block has two layers.

First, three kinds of “not a normal exit”.

• vmrun() returns a negative number: the VM does not exist, or the process was killed. We give up.
• e.entryfail is set: the entry into the guest failed, so the guest never ran. The CPU checks the guest state in the VMCS on every entry; reason 33 (EXIT_INVALID_GUEST) means a check failed, and qual says which one. This is almost always a bug in vmcsinit().
• EXIT_VMFAIL (0xFFFF, not a hardware reason): VMLAUNCH itself failed because the VMCS has invalid control or host fields. e.error is the “VM-instruction error number”. Look it up in SDM 33.4 (VM instruction error numbers).

Second, the real exits.

• EXIT_IO: an IN or OUT. doio() (Step 4) emulates the device, then continue goes to the next vmrun(). The kernel has already moved the guest’s RIP past the instruction, so the guest just keeps going.
• EXIT_HLT: the guest executed HLT. Our guest does that when it is done, so we treat it as normal termination and print the guest RIP. The kernel has also stepped past the HLT, so e.rip is the address of the instruction after it. That is why the final message says 0x10004E and not the address of the HLT itself. (If you change the guest code, this number changes. The expected value is for the handout’s helloguest.)
• exceptions, triple fault, EPT violation: the guest is broken, so we print everything we know and stop. The guest has no IDT, so the hypervisor intercepts all exceptions (the exception bitmap is all ones); the vector is in the low 8 bits of intrinfo (14 is a page fault, 13 a general-protection fault).

Anything else (a CPUID, a WRMSR the kernel does not emulate, …) falls out of the switch to the final printf. This monitor does not support it, and the message tells you exactly which exit reason to look up in SDM Appendix C.

Pitfall. continue inside the switch continues the enclosing for(;;), which is what we want. A break there would end the switch and fall into the “unhandled exit” message.

Step 4: Emulate the serial port

SDM: Table 30-5 “Exit Qualification for I/O Instructions” (the kernel decodes this for you); 28.1.3 “Instructions That Cause VM Exits Conditionally” (why IN/OUT exit at all). For the device: the “16550” UART datasheet (not part of the SDM). The guest-side IN/OUT semantics are in Vol. 2, the IN and OUT instruction pages.

This is the body of doio() in runvm.c:

  char c;

  /* OUT to the UART's transmit register: print the byte on our console
   * (file descriptor 1). This is how the guest's text reaches the user. */
  if(e->io.port == COM1 && !e->io.in){
    c = e->io.data;
    write(1, &c, 1);
    return;
  }
  /* IN from the line status register (COM1+5): a real driver polls this
   * until the "transmitter empty" bits (0x20 THRE, 0x40 TEMT) are set.
   * We are always ready to accept a byte, so report both bits. */
  if(e->io.port == COM1 + 5 && e->io.in){
    completein(vm, e->io.size, 0x60);  // line status: transmitter empty
    return;
  }
  /* Any other IN targets a port with no device behind it. Real buses
   * return all ones for such reads; OUTs to unknown ports are ignored. */
  if(e->io.in)
    completein(vm, e->io.size, 0xFFFFFFFF);  // no device: reads float high

On a PC, the first serial port (“COM1”) lives at I/O port 0x3F8. A 16550-style UART has a few byte-wide registers at consecutive ports. Two matter to us:

The three cases:

1. OUT to COM1. The guest is sending a character. The kernel already put the value in e->io.data (the low bytes of RAX, masked to the access size). We write that one byte to file descriptor 1, our own console. That is how the guest’s text reaches your terminal.
2. IN from COM1+5. A real serial driver reads the line status register and waits until the “transmitter holding register empty” bit (0x20) and the “transmitter empty” bit (0x40) are set before sending the next byte. We are always ready, so we answer 0x60, both bits set. Our hello guest does not poll (it just writes), but a more realistic guest, such as xv6 as a guest later, does.
3. Any other IN. There is no device. On a real bus, a read from a port nobody answers returns all ones, so we return 0xFFFFFFFF (completein() will keep only the bytes the guest asked for). An OUT to an unknown port is silently ignored: it falls through all three ifs and returns.

For an IN, notice that doio() never returns a value: the result goes into the guest’s RAX through completein(), because that is where the IN instruction puts it. That function is yours (next step).

Step 5: Your turn: completein()

Reference: Vol. 1, the section “General-Purpose Registers in 64-Bit Mode” (in the chapter “Basic Execution Environment”); Vol. 2, the IN instruction page; and the VMREG_* list in vmxapi.h.

The stub in the handout is

static void
completein(int vm, int size, uint val)
{
}

Goal. Make the guest see val as the result of an IN of size bytes (1, 2 or 4). The guest executed something like inb %dx, %al, inw %dx, %ax or inl %dx, %eax. When the guest resumes at the next vmrun(), its RAX must hold what that instruction would have left there.

The kernel does not complete the IN for you. It stepped RIP past the instruction and returned to you, so the destination register is yours to fill in, using vmgetreg() and vmsetreg() (system calls that read and write guest registers; the register codes are VMREG_* in vmxapi.h).

Hints.

• An IN writes only as many bytes as its operand size: AL (1 byte), AX (2 bytes) or EAX (4 bytes).

• The x86-64 rule for partial writes to a general-purpose register:

  • a 32-bit write (to EAX) zero-extends into the upper 32 bits of RAX, so the upper half becomes zero;
  • an 8-bit or 16-bit write (to AL or AX) leaves the other bytes of RAX unchanged.

  This is in Vol. 1 of the manual, in the section on general-purpose registers in 64-bit mode. (Volume 3, which this handout otherwise cites, does not repeat it.)

• So for 1 and 2 bytes you need the old value of RAX first. For 4 bytes you do not.

• val is a uint (32 bits) and may have junk in bits above size bytes. For example, the “no device” case passes 0xFFFFFFFF even for a 1-byte IN. Decide which bits of val to keep.

• vmgetreg(vm, reg, &x) stores into a 64-bit variable, so x must be a uint64.

What not to do. Do not just write val into RAX for every size. It looks right when you test with a 4-byte IN, and corrupts the upper bytes of RAX for 1- and 2-byte ones.

How to test. The handout’s helloguest never executes an IN, so the end-to-end run in the Checkpoint does not exercise this function. To test it, in a scratch copy of the tree add this to guestmain() in helloguest.c and print the result through the existing outb:

static inline unsigned char
inb(unsigned short port)
{
  unsigned char data;
  asm volatile("inb %1, %0" : "=a" (data) : "Nd" (port));
  return data;
}

Call inb(0x3F8 + 5) and print a character that depends on the result (for example, '0' + (inb(0x3F8 + 5) >> 5); with the right answer 0x60 >> 5 is 3). Then try a port with no device and check that you get 0xFF. Revert the change afterwards.

Step 6: The system-call plumbing you do not write

SDM: none. This is xv6 (Chapter 3 of the xv6 book, “System calls”).

Everything runvm calls (vmcreate, vmsetreg, vmgetreg, vmrun) is a system call, and all the plumbing is given. You do not write it, but you should be able to read it. A new xv6 system call needs the same five or six edits every time. Here is vmgetreg as the example:

The handler in sysvmx.c takes no C arguments. User space put them in registers and the trap handler saved them in myproc()->tf, so the handler fetches them with argint(), argint64() and argptr():

// int vmgetreg(int vm, int reg, uint64 *val)
//
// Read one register of VM number 'vm' into *val, using the same reg
// codes as vmsetreg(). Returns 0, or -1 if the VM or reg is bad or
// val is not a valid user pointer. The value is as of the last exit.
//
// val is an output pointer, so argptr() checks that all 8 bytes lie
// inside the process's address space before vmgetreg() stores to it.
uint64
sys_vmgetreg(void)
{
  int vm, reg;
  char *val;

  if(argint(0, &vm) < 0 || argint(1, &reg) < 0 ||
     argptr(2, &val, sizeof(uint64)) < 0)
    return -1;
  return vmgetreg(vm, reg, (uint64*)val);
}

Two points to notice, because they are the security-relevant part.

• argptr() checks a user pointer. The third argument is a pointer into the caller’s memory that the kernel will store 8 bytes through. If the kernel did not check it, a malicious program could pass a kernel address and make the kernel overwrite its own memory. argptr checks that the whole block, start and end, lies inside the process. vmrun() does the same for the whole struct vmexit.
• vmsetreg() takes a 64-bit value and so uses argint64(). argint() would silently truncate a guest RIP or RSP to 32 bits.

The function is small; the rule is the point: validate every argument that came from user space before using it.

Step 7: The guest starts in 32-bit protected mode

SDM: Vol. 3A 12.8.5 “Initializing IA-32e Mode”; 28.6 “Unrestricted Guests”; 27.6.2 “Processor-Based VM-Execution Controls” (the “unrestricted guest” control).

Open helloguestasm.S. It is the first code the guest runs, at VM_LOAD_ADDR. Its job: take the CPU from the state the hypervisor gives it to the state C code needs (64-bit mode, with a stack).

What state does the guest start in? A real PC starts in 16-bit real mode, and a boot loader (BIOS, GRUB) eventually hands the kernel a 32-bit protected-mode CPU with paging off and flat segments. xv6-64’s entry.S expects exactly that. Our hypervisor has no BIOS and no boot loader, so vmcsinit() writes that state straight into the VMCS:

• CR0 = PE | NE | ET: protected mode on, paging off;
• CS = selector 0x08 (flat 32-bit code), DS/ES/SS/FS/GS = selector 0x10 (flat data);
• the GDTR in the VMCS is empty (base 0, limit 0);
• RFLAGS = 2 (only the always-one bit), so interrupts are off;
• RIP = VM_LOAD_ADDR, set by runvm; RSP is not set.

Protected mode with paging off needs the VMX feature unrestricted guest. In the first VMX processors, CR0.PE and CR0.PG had to be 1 in VMX operation, which would have forced the guest to have paging on before it could set up any page table. SDM 28.6 explains that with “unrestricted guest” set, both may be 0.

The empty GDTR is not a mistake: the descriptors named by the segment registers are cached inside the CPU (they were loaded from the VMCS guest-state area), so the guest can run with an empty GDT as long as it does not reload a segment register. It will, so it loads its own GDT first (step 4 below).

Step 8: Step by step into 64-bit mode

SDM: Vol. 3A 12.8.5 “Initializing IA-32e Mode” (the sequence); 3.4.5 “Segment Descriptors” (the L flag); Table 5-15, 5-17, 5-18 (paging entry formats, Vol. 3A chapter 5); 5.5.2 “Use of CR3 with Ordinary 4-Level Paging”.

The sequence in helloguestasm.S follows SDM 12.8.5 exactly. The manual says IA-32e mode (“long mode”) needs physical-address extensions (PAE) and 4-level paging, CR3 pointing at the PML4, EFER.LME set, and then paging turned on.

The code below is the file’s own code, in the order the CPU executes it. You do not write it; read it with the SDM open next to it.

  # Step 1: set CR4.PAE (Physical Address Extension). CR4 cannot be
  # modified directly, so read-modify-write through EAX.
  # (CR4.VMXE is host-owned: the guest reads it as 0, so it is not
  # disturbed by this write.)
  movl    %cr4, %eax
  orl     $(CR4_PAE), %eax
  movl    %eax, %cr4

1. CR4.PAE = 1. Long mode only works with PAE-format page tables. If you set CR0.PG later with EFER.LME set and PAE clear, the CPU raises #GP (SDM 12.8.5, “Failure to enable PAE will result in a #GP fault”). CR4 cannot be modified directly, so the code does a read-modify-write through EAX. The guest reads CR4.VMXE as 0 (the host owns it), so this write does not disturb VMX.

  # Step 2: point CR3 at the top-level page table (PML4), defined
  # below. Its address is a link-time constant that fits in 32 bits
  # because the guest is loaded low (0x100000 plus a few KB). Nothing
  # is translated yet: paging is still off.
  movl    $pml4, %eax
  movl    %eax, %cr3

2. CR3 = pml4. The top-level page table is static data further down in the same file. Its address is a link-time constant, and it fits in 32 bits because the guest is loaded low (0x100000 plus a few KB). That matters: SDM 12.8.5 says MOV CR3 is executed in legacy mode, so only the low 32 bits are written and the PML4 must be in the first 4 GB. Paging is still off, so nothing is translated yet.

  # Step 3: set EFER.LME (long mode enable). EFER is a model-specific
  # register, accessed with RDMSR/WRMSR: ECX selects the MSR, and the
  # 64-bit value is in EDX:EAX. Read it, set the bit, write it back.
  #
  # The hypervisor does not give the guest direct access to MSRs
  # (there is no MSR bitmap), so both RDMSR and WRMSR cause a VM exit.
  # vmx.c (vmxmsr) emulates them: it keeps the guest's EFER in
  # struct vm, copies it into the VMCS before entering the guest, and
  # advances RIP. To the guest, the exits are invisible; the guest just
  # sees the instructions complete. The hypervisor also sets the
  # VM-entry control "IA-32e mode guest" later, when the guest's EFER
  # has LMA set, so that the processor state stays consistent.
  movl    $EFER_MSR, %ecx
  rdmsr                               # EDX:EAX = EFER (exits, emulated)
  orl     $EFER_MSR_LME, %eax
  wrmsr                               # EFER = EDX:EAX (exits, emulated)

3. EFER.LME = 1. EFER is a model-specific register, accessed with RDMSR/WRMSR: ECX selects the MSR and the 64-bit value is in EDX:EAX. The code reads it, sets bit 8 (LME), and writes it back.

In this guest, both instructions cause a VM exit: the hypervisor gives the guest no MSR bitmap, so every RDMSR and WRMSR traps. The kernel (vmxmsr() in vmx.c) emulates them, keeping the guest’s EFER in struct vm, and steps past the instruction. These exits never reach runvm. From the guest’s point of view, the instructions simply complete. (The kernel also has to set the VM-entry control “IA-32e mode guest” at the right moment, which is why EFER is emulated instead of passed through.)

  lgdt    gdtdesc

4. Load a GDT. The VMCS gave the guest an empty GDT, and the far jump below needs a valid 64-bit code descriptor, so the guest loads its own. gdtdesc holds a 16-bit limit and a 32-bit base (32 bits because lgdt runs in 32-bit mode). The GDT has three 8-byte entries; a selector is the byte offset of the entry:

The L bit (bit 21 of the second doubleword, SDM 3.4.5) says “the instructions in this code segment run in 64-bit mode”. When L is 1, the D bit must be 0, which is why the code descriptor has flags 0x20 and not 0x60. In 64-bit mode, base and limit are ignored.

  # Step 5: set CR0.PG to turn on paging. Because CR4.PAE, CR3 and
  # EFER.LME are all set, this activates long mode: EFER.LMA becomes 1
  # (the processor sets it; the hypervisor preserves it). The
  # instruction after this one is still fetched from the same address
  # because the page tables identity-map it.
  movl    %cr0, %eax
  orl     $(CR0_PG), %eax
  movl    %eax, %cr0

5. CR0.PG = 1. With PAE, CR3 and EFER.LME all in place, turning on paging activates long mode: the CPU itself sets EFER.LMA (“long mode active”, SDM 12.8.5 step 5). The processor is now in 32-bit compatibility mode, because CS still names a 32-bit descriptor. The instruction after this one is fetched through the new page tables, which is fine because they identity-map it (virtual address = physical address). SDM 12.8.5 requires this explicitly: the MOV CR0 and the following instructions must be in an identity-mapped page.

  ljmp    $8, $start64

6. Far jump to the 64-bit code segment. A far jump reloads CS with selector 8, whose descriptor has L = 1, so the CPU switches to true 64-bit mode and continues at start64. This is the only way to change CS’s mode; MOV to CS is not allowed.

The static page tables. They are in the .data section, aligned to 4 KB, and they identity-map the first 1 GB with 2 MB pages:

The flag bits are 0x1 present, 0x2 writable, and, in the pd only, 0x80 PS (“page size”: this entry maps a 2 MB page instead of pointing to a lower table; SDM Table 5-18, bit 7). With PS set there are no level-1 tables. (i << 21) is i times 2 MB, so the 512 entries map physical 0 to 1 GB onto themselves. The assembler computes them with a .rept 512 loop. The guest has at most 64 MB (VM_MAXMEM), so this covers its whole memory and more.

These tables contain absolute addresses of labels in the file (pdpt, pd, gdt). They are only correct if the program was linked at the address it actually runs at (Step 9).

start64:
  # Load the data segment registers with selector 16, the data
  # descriptor. In 64-bit mode DS/ES/SS bases and limits are ignored,
  # but they should hold a valid selector (a null or stale one in SS
  # can fault). FS/GS are left alone; they are unused here.
  movw    $16, %ax
  movw    %ax, %ds
  movw    %ax, %es
  movw    %ax, %ss
  # The stack: set RSP to the top of the 4 KB reserved in .bss below
  # (stacks grow down). Now C code can run.
  movq    $stacktop, %rsp
  # Run the C program (helloguest.c). It prints through the serial port.
  call    guestmain
  # guestmain returned: halt. HLT causes a VM exit that runvm.c reports
  # as "guest halted". The loop guards against the guest being resumed.
1:
  hlt
  jmp     1b

7. The 64-bit side. Load DS, ES and SS with selector 16 (in 64-bit mode their bases and limits are ignored, but the selector should be valid; a stale one in SS can fault). Set RSP to the top of the 4 KB stack reserved in .bss, since stacks grow down. Now C code can run: call guestmain. If it returns, HLT causes the VM exit that runvm reports as “guest halted”.

Step 9: The guest program, the linker script, and the image

SDM: Vol. 2, the OUT instruction page; Table 30-5 for how the hypervisor sees it.

helloguest.c. guestmain() is the whole program:

  // Send each character of the message, up to the terminating NUL,
  // to COM1. A real driver would first poll the line status register
  // (port COM1+5) for "transmitter empty"; runvm always says yes, so
  // we skip that and just write.
  for(s = "Hello from a VT-x guest\n"; *s; s++)
    outb(COM1, *s);

outb() is one line of inline assembly: outb %al, %dx with the data in AL and the port in DX ("a" and "Nd" in the constraints). The OUT instruction is privileged I/O and, because the hypervisor sets “unconditional I/O exiting”, every OUT causes a VM exit. So printing 24 characters costs 24 round trips: guest, kernel, runvm, kernel, guest. A real guest would batch work in a buffer; we don’t care here. The volatile keeps the compiler from removing or reordering an instruction that has a side effect.

helloguest.ld. The linker script says . = 0x100000; and puts .text.start (the trampoline, label start) first. The comment in the file says KEEP IN SYNC, and it means it: 0x100000 must equal VM_LOAD_ADDR in vmxapi.h. The kernel copies the image to that guest-physical address, runvm jumps there, and the page tables and GDT contain addresses computed at link time. Change one without the other and the guest crashes, usually at the first ljmp or the first access through gdtdesc.

How a guest becomes a file in fs.img. Look at the helloguest rule in the Makefile:

helloguest: helloguestasm.S helloguest.c helloguest.ld
	$(CC) $(GUESTCFLAGS) -c helloguestasm.S -o helloguestasm.o
	$(CC) $(GUESTCFLAGS) -c helloguest.c -o helloguest.o
	$(LD) -T helloguest.ld -z max-page-size=0x1000 -o helloguest.elf helloguestasm.o helloguest.o
	$(OBJDUMP) -S helloguest.elf > helloguest.asm
	$(OBJCOPY) -O binary helloguest.elf helloguest

1. Compile the assembly and the C file. GUESTCFLAGS includes -ffreestanding -nostdinc: there is no libc and no kernel here. -mno-red-zone -mgeneral-regs-only keep the compiler from using features that need a stack or FPU state the guest never set up.
2. Link both with helloguest.ld into an ELF file, helloguest.elf.
3. objcopy -O binary strips the ELF headers and keeps only the loaded bytes, in address order, starting at 0x100000. The result, helloguest, is a flat binary: byte n of the file is meant to live at 0x100000 + n.
4. fs.img is built with ./mkfs fs.img README $(UPROGS) helloguest, so the flat binary is copied into the xv6 file system next to the user programs. runvm helloguest simply opens it by name.

helloguest.asm is the disassembly of the ELF. It is the best way to see where every instruction ended up. The 0x10004E that runvm prints is the address right after the hlt in that listing.

Checkpoint

You now have everything. Build and boot, then run the monitor.

On QEMU/KVM (CADE machines). Put your QEMU on the PATH, as you did for the earlier parts, and run:

$ make qemu-nox
...
cpu1: VMX enabled
cpu0: VMX enabled
...
$ runvm helloguest
Hello from a VT-x guest
runvm: guest halted at rip 0x10004E

This needs a VMX-capable Intel CPU with nested KVM. (make qemu-nox passes -enable-kvm -cpu host for that reason.)

On Bochs (any other host). Run make bochs-nox and type runvm helloguest at the shell, the same as with QEMU. Bochs is slow, so boot takes a while. Quit with Ctrl-A x.

What to check.

1. Both lines appear, with the exact address 0x10004E.
2. Run runvm helloguest 1 (1 MB of guest memory). The guest is loaded at 1 MB, so it cannot fit. Note what message you get and explain why in answers.md (Question 6.3).
3. With make qemu-nox QEMUKVM= (no VMX), runvm helloguest should fail cleanly with runvm: vmcreate failed, not panic.

If something fails:

Questions

Answer these in answers.md (in your repo; each question has a slot), two or three sentences each.

1. In doio(), why does an IN need the monitor to write the guest’s RAX with vmsetreg(), instead of doio() just returning a value? And why does completein() read RAX with vmgetreg() for a 1-byte IN but not for a 4-byte one?
2. helloguest.ld links the guest at 0x100000. Suppose you change it to 0x200000 but leave VM_LOAD_ADDR alone. Name two things in the guest that will go wrong and say why. (Look at the page tables, the gdtdesc, and where the first instruction is.)
3. The guest in Step 8 sets CR4.PAE before CR0.PG. What happens if you swap the order of these steps? Then, for the Checkpoint’s runvm helloguest 1 experiment: which message do you get, and which check in the kernel produces it?
4. Each OUT in the guest causes a VM exit, a trip through the kernel, and a return to runvm, which calls write(). Why did the design put the serial port in runvm and not in the kernel? What would you gain, and what would you lose, by emulating it inside the kernel?

Major design decisions

VMCLEAR after every exit. The VMCS is VMCLEARed after each exit, before popcli(). A VMCS may be cached by the CPU where it was loaded, and xv6’s timer can move a process to another CPU once interrupts are on. By clearing it, the VMCS is cached nowhere, and the next entry simply loads it on whatever CPU it runs. As a consequence, every entry is a VMLAUNCH, never a VMRESUME. A faster design would keep the VMCS loaded and handle migration with scheduler hooks, as KVM does with preempt notifiers. We chose simplicity.

Host state is written before every entry. CR3, the TR base, and the GDTR base depend on the CPU and the current process, so they cannot be set once.

KVM-style split. The kernel handles only what it must, and everything else is the monitor’s job. The kernel never opens files for the hypervisor: runvm reads the image and passes it in.

Unrestricted guest with a flat 32-bit start. The guest starts in protected mode with paging off, which avoids emulating real mode and lets the guest’s own trampoline switch to 64-bit mode, exactly as xv6 does on real hardware.

Exception bitmap of all ones. Without an IDT, a guest exception would become a triple fault. Intercepting all of them gives a debuggable report instead.

Port I/O exits unconditionally. There is no I/O bitmap; every IN and OUT exits to runvm, which emulates the device.

Completing an IN is done in user space. (Two designs were considered; the second is what the code does.)

• Alternative: vmrun() completes a pending IN: runvm leaves the value in exit.io.data, and the kernel merges it into RAX on the next vmrun(). This is what KVM does, and the kernel owns the x86 width rules (a 4-byte IN zero-extends into RAX; a 1- or 2-byte IN replaces only the low bytes). The cost is hidden state in the kernel and a contract that the next vmrun() reads from struct vmexit.
• Chosen: runvm writes RAX itself. After an IN exit, runvm reads RAX with vmgetreg(), merges the value into the low 1, 2 or 4 bytes, and stores it with vmsetreg() before the next vmrun(). The kernel keeps no pending-IN state. All register changes go through one visible interface, and vmgetreg gives user space control over all the registers, not just RAX. The helper is completein() in runvm.c.

vmgetreg returns values as of the last exit. It reads the software copy in struct vm. It does not VMPTRLD and VMREAD, which would need pushcli, VMPTRLD, VMREAD, VMCLEAR and popcli for each read.

Pitfalls

• VMCLEAR must happen on the CPU where the VMCS was loaded. Keep the clear-before-popcli() invariant.
• Never read VMX capability MSRs without checking for VMX first: on hardware without VMX the read faults and panics the kernel.
• kalloc() does not zero pages. Zero guest memory and EPT tables.
• After an exit caused by an instruction, the CPU does not advance RIP. The hypervisor must do it (vm->rip += inslen) when it emulates the instruction.
• %x in cprintf reads a 64-bit value; cast 32-bit arguments to uint64.
• The guest is a flat binary linked at VM_LOAD_ADDR. Keep vmxapi.h and helloguest.ld in sync.

What to hand in

For the whole homework:

• Your source code for all parts, with the stripped function bodies filled in. Do not change the given code, the linker script, or vmxapi.h.
• answers.md: your answers to the Questions at the end of every part (Parts 1 to 6). The repo already has a template with every question and a slot for each answer, numbered by part and question (for example 1.2 or 6.4). Fill in your name and environment at the top, and two or three sentences for each answer.

A grader should be able to do the following with your tree: build it, boot it (QEMU/KVM, or Bochs with the init change), and see

cpuN: VMX enabled            (one line for each CPU, in any order)
Hello from a VT-x guest
runvm: guest halted at rip 0x10004E

After runvm exits, the shell should still work, so they can run runvm helloguest a second and third time and get the same output. (If the second run fails, look at how a VM is released when its process exits, and at the VMCLEAR rule in the run loop.)

Where this goes next

What you built is a complete, if tiny, hypervisor: the same structure as KVM plus QEMU, with a guest that does nothing but print. The longer goal of this series of homeworks is to run xv6 itself as the guest. To get there, later assignments add a virtual timer (the guest’s local APIC timer, so that the guest’s scheduler runs), interrupt injection (the kernel arranging for a virtual interrupt to be taken when the guest resumes), an emulated IDE disk so that the guest can read its own file system image, and the handling of more exits (more MSRs, CPUID, control registers). Each one follows the pattern you used here: configure the VMCS so the guest’s action traps, decide whether the kernel or runvm emulates it, step past the instruction, and resume.