Debugging the host: gdb, VS Code and Bochs

This page applies to the xv6-vmx homework series. It explains how to attach a debugger to xv6 while it runs under QEMU or Bochs.

When the kernel panics or the guest does something strange, a debugger helps more than cprintf. You debug the host, that is xv6 with your hypervisor, not the guest. You cannot single-step into the guest: after VMLAUNCH the guest runs on its own, and the debugger sees the host again only at the next VM exit. So the useful breakpoints are in the kernel: vmxinit, vmcreate, vmrun, vmx_enter, vmx_exit, vmxhandle.

gdb with QEMU

This is the best setup, and it works on the CADE lab machines. Use two terminals, both in the top directory of the repo.

  1. In the first terminal, start QEMU so that it waits for gdb:

    $ make qemu-nox-gdb CPUS=1

    QEMU starts paused (-S) and listens for gdb on a TCP port that depends on your user id (uid % 5000 + 25000), so that students on the same machine do not collide. The Makefile writes the port into a file called .gdbinit. CPUS=1 is optional; it makes debugging simpler, because each CPU is shown as a separate gdb thread.

  2. In the second terminal, start gdb:

    $ gdb

    gdb reads .gdbinit, which loads the symbols of kernel, and connects to QEMU. If gdb says that auto-loading .gdbinit was declined, start it with gdb -nx -x .gdbinit instead. On a Mac, run gdb inside nix develop; it is already set up for x86-64.

  3. At the (gdb) prompt, QEMU is stopped at the very first instruction. Set breakpoints and continue:

    (gdb) break vmxinit
(gdb) continue
Thread 2 hit Breakpoint 1, vmxinit () at vmx.c:212
(gdb) bt
(gdb) next

    Commands you will use all the time: break FUNCTION or break FILE:LINE, continue, next (step over), step (step into), bt (call stack), print EXPRESSION (use print/x for hex), info registers, x/8gx $rsp (dump 8 words of the stack), info threads and thread N (switch CPU), and quit. Stop QEMU with Ctrl-a x in the first terminal.

Some hypervisor-specific tips:

gdb’s text interface (TUI)

gdb has a built-in full-screen mode that shows the source code, the registers or the disassembly above the command line, and follows you as you step. It works over ssh and in any terminal, so it is a good middle ground between plain gdb and VS Code. Start gdb with gdb -tui, or switch it on at any time with Ctrl-x a (press Ctrl-x, release, press a; the same keys switch it off again).

$ gdb -tui
(gdb) break vmx_enter
(gdb) continue

Useful TUI commands:

Command or keys What it does
layout src source code above the command window
layout asm disassembly; use it in vmxasm.S, where you want to watch each instruction
layout split source and disassembly together
layout regs registers above the source or the disassembly; changed registers are highlighted, which is handy in vmx_exit
Ctrl-x 1, Ctrl-x 2 one window, or cycle through two-window layouts
Ctrl-x o move the keyboard focus to the next window (focus cmd and focus src also work)
Ctrl-l redraw the screen if QEMU’s output or an error message messed it up
tui disable back to the plain command line

The window with the focus gets the arrow keys. When the source window has the focus, the arrows scroll the file; when the command window has it, they move through your command history. If the arrow keys do not do what you expect, press Ctrl-x o.

QEMU’s own output goes to the first terminal, not to the TUI, so the screen stays clean. The TUI works with QEMU only. The Bochs debugger is a separate program with its own text prompt.

gdb in VS Code

If you prefer a graphical debugger, you can use the same QEMU setup from VS Code.

  1. Install the Native Debug extension (publisher WebFreak) in VS Code, and make sure that gdb is on the PATH of the terminal that starts VS Code. On a Mac, run nix develop first and start VS Code from that shell with code ..
  2. In the repo, run make launch.json. It writes .vscode/launch.json, with an “Attach to QEMU” configuration that uses your port, and a file called .gdbinit.vscode.
  3. In a terminal, run make qemu-nox-gdb CPUS=1.
  4. In VS Code, open Run and Debug, choose Attach to QEMU, and press the green arrow. The program stops at the first instruction; press Continue (F5). Click in the left margin of a source file to set a breakpoint. The Debug Console takes gdb commands.

Debugging on a CADE machine from your own computer

If you run QEMU on a CADE lab machine but want to use VS Code or gdb on your own laptop, there are two ways. We have not tested either one on the lab machines, so tell us if something does not work.

Option 1: VS Code Remote-SSH (simplest)

  1. Install the Remote - SSH extension in VS Code, and connect to the lab machine (use a jump host if you normally reach CADE through a gateway).
  2. Open the repo folder on the lab machine, and install Native Debug in the remote window when VS Code offers to.
  3. Everything now runs on CADE. Follow the steps of “gdb in VS Code” above: make launch.json, make qemu-nox-gdb CPUS=1 in the integrated terminal, and Attach to QEMU.

The kernel, the source files, the gdb and the port are all on the same machine, so nothing can get out of step. VS Code installs a small server in your home directory on the lab machine; if your quota or CADE’s rules do not allow that, use option 2.

Option 2: local VS Code and gdb through an SSH tunnel

  1. On the lab machine, build and start QEMU as usual: make qemu-nox-gdb CPUS=1. Its gdb port is uid % 5000 + 25000 for your CADE user id. Compute it with expr $(id -u) % 5000 + 25000.

  2. On your computer, forward that port (replace PORT and the machine name; add -J jumphost if you need one):

    $ ssh -L PORT:localhost:PORT your-lab-machine

    Leave this connection open.

  3. On your computer, in your own copy of the repo, inside nix develop if you are on a Mac: make launch.json GDBPORT=PORT, then use Attach to QEMU in VS Code, or run gdb yourself after target remote localhost:PORT.

Your local kernel file must be exactly the one that runs on CADE. gdb takes the symbols and the source line numbers from the file on your computer, but the code runs on the lab machine. If you changed a source file and rebuilt in only one place, the breakpoints land in the wrong places or at nonsense addresses. Rebuild in both places, or copy the kernel file from the lab machine to your computer each time (scp your-lab-machine:path/to/kernel .).

QEMU’s gdb port listens on every network interface of the lab machine, not only on localhost, so another user on the network could connect to it while it is open. Stop QEMU (Ctrl-a x) when you finish debugging.

Bochs

The Bochs from Nix is not built with gdb support (it says Bochs is not compiled with gdbstub support), so gdb cannot attach to it. It has its own debugger instead, which is enough to stop in the kernel and look at registers and the stack. It does not know your symbols, so you look up the address of a function in kernel.sym:

$ grep -w vmxinit kernel.sym
0000080000109bf8 vmxinit

Write the debugger commands in a file, for example dbg.rc:

lb 0x0000080000109bf8
c
r
bt
q

Here lb sets a breakpoint at a linear (virtual) address, c continues, r prints the registers, bt prints a call stack, and q quits. Then, inside nix develop, after make fs.img xv6.img:

$ bochs -q -f dot-bochsrc-nox -debugger -rc dbg.rc

Bochs stops at the breakpoint and runs your commands. When it hits it, the first number in parentheses, (1), is the CPU that stopped. Type help at the <bochs:N> prompt to see all the debugger commands. If you leave out -rc dbg.rc, you get that prompt right away, but the xv6 console and the debugger then share the terminal, which is awkward. The script file is easier. Delete fs.img.lock and xv6.img.lock if Bochs complains after being killed.