CS 6465 - Advanced OS Implementation

cs6465 teaches advanced topics in operating systems through a combination of hands-on engineering approach and studying implementation of modern full-featured kernels like Linux, virtualization stacks like Xen and KVM, various operating system organizations like unikernels and microkernels, etc.

This class is for students who plan to stay in the systems field, not just building new OS kernels and hypervisors, but working on parts of modern cloud and datacenter infrastructure, be an expert in low-level systems security, be able to work on cycle-level code optimizations for warehouse scale AI and big data systems, etc. I.e., you will have a rare skill to work with the teams developing macOS and iOS kernels, XNU, Windows and Linux kernels, novel hypervisors like Amazon Nitro and Firecracker, etc. This is a rare and potentially well-compensated skill.

The idea is that you can learn enough about modern commodity operating systems like Linux, hypervisors like KVM, Xen and VMware, container technologies and a range of cutting edge research topics that you can either get a job in this area or start working on competitive research.

You will study, in detail, organization of modern commodity kernel like Linux, commodity hypervisors like KVM and Xen, understand performance and security problems of modern operating systems. You will further understand major research directions in the areas of security, performance and reliability of operating systems. We will cover ideas of microkernels, exokernels, unikernels and library kernels, ideas of retrofitting isolation into commodity kernels, modern hardware isolation mechanisms, software fault isolation and WASM, basics of software verification and its potential for development of formally correct operating systems.

To master the concepts, cs6465 is organized as a series of in-class presentations that most often take a deep dive into organization of one of the modern mucrokernels, hypervisors, or kernels and often will require reading of a research paper. The homeworks will follow the lectures, e.g., read through the internals of the Linux memory allocator, etc., and will be followed by a project (see list of possible projects) for ideas.

Approximate schedule for Fall 2026 (with lectures and recordings from 2024)

Aug 19
Lecture 01 - Linux Memory Management (video)

Aug 21
Lecture 01 - Linux Memory Management (part 2) (video)

Aug 26
Lecture 02 - Linux Process Memory Management (video)

Aug 28
Lecture 02 - Linux Process Memory Management (part 2) (video)

Sept 2
No class (Labor Day Holiday)

Sept 4
Lecture 02 - Linux Process Memory Management (part 3) (video)

Sept 9
Lecture 03 - Linux Buffer Cache (video)

Sept 11
Lecture 04 - Recap of Linux Virtual Memory Subsystem (video)

Sept 16
Lecture 05 - Linux Device Drivers (video)

Sept 18
Lecture 06 - Synchronization (video) Reading: Non-scalable Locks Are Dangerous

Sept 23 No class (Anton is traveling)

Sept 25 Lecture 06 - Synchronizatioin (part 2) (video Reading: RCU Usage In the Linux Kernel: One Decade Later

Sept 30 Lecture 06 - Synchronizatioin (part 3) (video)

Oct 2 Lecture 07 - Virtualization (video) Reading: A Comparison of Software and Hardware Techniques for x86 Virtualization

Oct 7 No class (Fall Break)

Oct 9 No class (Fall Break)

Oct 14 Lecture 07 - Virtualization (part 2) (video)

Oct 16 Lecture 07 - Virtualization (part 3, hardware supported virtualization) (video)

Oct 21 Lecture 07 - Virtualization (part 4, efficient I/O) (video) Reading: Safe Hardware Access with the Xen Virtual Machine Monitor

Oct 23 Lecture 07 - Virtualization (part 5, migration, checkpoints, replication) (video) Reading: Parallax: Virtual Disks for Virtual Machines

Oct 28 Class project discussions (video)

Oct 30 Lecture 08 - Kernel Vulnerabilities (video) Reading: SoK: Eternal War in Memory

Nov 6 Lecture 08 - Kernel Vulnerabilities (part2) (video)

Nov 11 Lecture 09 - Microkernels and Exokernels (video)

Nov 13 Lecture 09 - Microkernels and Exokernels (video) Lecture 10 - Modern Isolation Mechanisms

Nov 18 Lecture 10 - Modern Isolation Mechanisms (video)

Nov 20 Lecture 10 - Modern Isolation Mechanisms (part 2) (video)

Nov 25 Lecture 11 - Bringing Isolation into the Kernel (video)

Nov 26 Lecture 12 - Safe Operating Systems (video)

Dec 2 Lecture 12 - Safe Operating Systems (video) Lecture 13 - Isolation in the Field

Dec 4 Lecture 14 - Performance, Containers, Virtualization (video)