Instructions for setting up xv6-64 on CADE servers

Xv6-64 is a real operating system kernel, and hence, it needs real hardware to boot. Fortunately, today we can emulate hardware in software. Programs like QEMU can emulate functionality of the real physical CPU in software. I.e., QEMU implements the normal CPU loop similar to the one we discussed in class: fetches an instruction pointed by the instruction pointer register (RIP), decodes it, performs all permission and condition checks, computes the outcome, increments RIP and continues to the next instruction. Like a real PC platform, QEMU emulates hardware boot protocol. QEMU starts by loading the disk sector number 0 into the memory location 0x7c00 and jumping to it. Xv6-64 takes it from there. At a high level, for xv6-64 it does not matter if it runs on the real hardware or under QEMU. Of course, emulation is slower than real hardware, but besides that as long as QEMU implements the logic of the CPU correctly we do not see any deviations from a real baremetal execution. Surprisingly, QEMU is reasonably fast, so you as a human barely notice the difference.

SSH into CADE

To configure your xv6-64 environment, login into one of CADE servers, e.g., using the campus VPN, pick a random server by replacing XX with a random number from 1 to 40 and run the following command in your terminal:

ssh <your-uid>@lab1-XX.eng.utah.edu

Make sure Qemu is in your path (Use bash for all the following commands)

We have build a version of Qemu emulator for you. You only need to add it to your path. You can either add it every time you log in to the CADE machine by exporting the PATH variable, or you can add this line to .bashrc file in your home folder.

export PATH=$PATH:/home/cs5460/qemu-install/bin

If you decide to add the PATH permanently to your .bashrc the line should look something like

export PATH=/home/cs5460/qemu-install/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:$HOME/bin:$PATH

Add this line to your ~/.bashrc file and then restart bash (this step is necessary if you are on CADE)

Clone, build, and boot xv6

If you haven’t done this, create cs5460 folder for all projects in this class.

cd ~/
mkdir cs5460

Clone this xv6-64 repository adapted for this class into the cs5460 folder.

cd ~/cs5460
git clone https://github.com/mars-research/xv6-64.git

After the repository has been cloned, go ahead and build xv6-64. You will see a bunch of output from the make command that invokes the gcc compiler on all kernel files, links the kernel with ld, creates the kernel file system with all user-level programs that will be available inside xv6-64, and so on.

cd xv6-64
make

Now you’re ready to run xv6!

make qemu-nox

This is a fork of xv6-64, which is inturn a fork of the official x86 version of xv6 by MIT-PDOS, combined with some of the modifications from NYU’s fork of xv6 which allow easy debugging with vscode. Feel free to explore the other repositories, but for the labs use the one from https://github.com/mars-research/xv6-64.

You will need this VSCode Extension (Native Debug) to integrate QEMU-GDB with VSCode.

You are now running xv6 on top a hardware platform that is emulated by the QEMU emulator. You are now done with the xv6-64 setup and can continue moving to any homeworks that are currently assigned. To exit xv6-64 running under QEMU you can terminate it with Ctrl-A X.

You can find more information about QEMU monitor and GDB debugger here, feel free to explore them.

Attaching gdb

First, in the same folder where you have xv6-64 installed, run the following command to create a GDB initialization file.

make .gdbinit

If you can boot into xv6 your environment is working and you can debug your xv6 system with gdb. From inside your the same folder, run the following command to start xv6-64 with gdb support:

make qemu-nox-gdb

Open another shell, enter nix shell in the same folder where xv6 is installed and attach gdb to it:

gdb

Note, you might add something like

add-auto-load-safe-path /<where-did-you-put-it>/xv6-64/.gdbinit

to the .gdbinit in your home folder like gdb suggests.

Now you can set brakepoints and debug xv6, e.g., set breakpoint on main, the instruction at the boot sector (0x7c00) and then continue execution:

(gdb)b main
(gdb)b *0x7c00
(gdb)c

Note: it is a known issue that qemu-system-x86_64 and gdb do not play well together in terms of specifying the debug target architecture, so the debug target is forced to be i386:x86-64, no matter what mode (16-bit real mode, 32-bit protected mode, or 64-bit long mode) QEMU is actually running in. Because of this, if you disassemble the code, the assembly instructions will be disassembled as 64-bit instructions and the registers will be shown as 64-bit registers, even when the code is actually running in 16-bit real mode or 32-bit protected mode. However, according to our experience, this does not make debugging more difficult.

Alternative ways to install xv6 if you feel like it (not required for this class)

Xv6 with Nix (works on ARM Mac and Intel Linux systems)

On the new ARM-based CPUs you need to cross-compile everything to get the x86 qemu running. The easiest way to do this is to use the Nix environment we provide:

git clone https://github.com/mars-research/xv6-64.git

Install Nix (follow the Nix installation instructions online: (https://nixcademy.com/posts/nix-on-macos/)[https://nixcademy.com/posts/nix-on-macos/], then enter inside the cloned repository

cd xv6-64
nix develop

This may take a while the first time. After it finishes, you can now build xv6 inside the Nix shell.

make

Now you’re ready to run xv6!

make qemu-nox