# Intel VMX at Runtime: Why VM Entry, Guest Execution, and VM Exit Work the Way They Do Chapters 26 and 27 describe the machinery: VMX operation, VMCS regions, state fields, execution controls, and transition controls. Chapters 28 through 30 make that machinery move. They answer three runtime questions: 1. While the guest is running, which operations execute natively and which return control to the hypervisor? 2. How does the processor safely construct a guest context during VM entry? 3. How does it stop the guest, explain what happened, and construct a safe host context during VM exit? The most useful mental model is not “the CPU switches into an emulator.” The guest normally runs directly on the processor. VMX adds policy checks at carefully chosen architectural boundaries. ```mermaid flowchart LR A["Host prepares VMCS"] --> B["VMLAUNCH or VMRESUME"] B --> C["VM entry validates and loads guest"] C --> D["Guest executes natively"] D --> E{"Exit condition?"} E -->|No| D E -->|Yes| F["VM exit saves guest and loads host"] F --> G["Host handles cause"] G --> A ``` This loop is the center of hardware virtualization. Performance depends on keeping the native-execution portion wide, while correctness depends on making each boundary precise. --- ## Chapter 28: VMX non-root operation is native execution with escape hatches VMX non-root operation is not a reduced instruction interpreter. Most guest instructions use the ordinary execution engine, caches, branch predictors, page walkers, and arithmetic units. The important difference is that selected instructions and events are routed through VMX policy. Depending on the controls, an operation may: - execute with normal architectural behavior; - execute with VMX-modified behavior; - cause a fault inside the guest; - cause a VM exit before it executes; - complete and then cause a VM exit; - be handled by a dedicated virtualization facility without an exit. That range of outcomes is deliberate. “Always trap” would be simple but slow. “Always execute” would be fast but unsafe. VMX turns individual architectural surfaces into independently selectable boundaries. ## Fault-like and trap-like VM exits Most instruction-induced VM exits are **fault-like**: ```text guest reaches intercepted instruction -> VM exit occurs before the instruction executes -> saved guest RIP points at the intercepted instruction -> hypervisor emulates, rejects, or changes policy ``` For example, if `CPUID` exits, the guest has not yet received CPUID results. The VMM can synthesize values in the guest register frame, advance guest `RIP` by the reported instruction length, and resume. Some exits are **trap-like**: the triggering operation completes before the exit. Saved state therefore reflects its effects, and guest `RIP` normally identifies the following instruction. Examples include exits caused after certain debug conditions, APIC-write emulation, bus-lock detection, and some virtualized control-register or x2APIC operations. **Why the distinction matters:** A handler must know whether it is completing an operation that never ran or reacting to an operation that already changed architectural state. Re-executing a trap-like operation can duplicate side effects; skipping a fault-like one can lose the operation entirely. ### Why use trap-like exits for anything other than debugging? Here, *trap-like* describes the architectural boundary captured by the VM exit; it does not mean that the exit exists primarily for a debugger. A non-debug trap-like exit is useful when the guest operation itself is safe to complete, but its result creates a condition that the VMM must handle. That gives VMX two different interception patterns: | VMM requirement | Appropriate boundary | |---|---| | Decide whether an operation is allowed, reject it, or emulate it with different results | Fault-like exit before execution | | Observe the completed result, respond to a newly reached threshold, or handle a consequence detected during execution | Trap-like exit after execution | The second pattern avoids making the VMM reproduce work that hardware can already do correctly. Hardware performs the common operation, then exits only when its result crosses a boundary selected by the VMM. #### Example: `CR8`, interrupt priority, and a TPR-threshold exit On x86-64, `CR8` represents the task-priority class of the local APIC. An APIC interrupt vector's high four bits define its priority class: ```text priority class = vector >> 4 vector 0x52 -> class 5 vector 0x91 -> class 9 ``` Ignoring other blocking conditions for the moment, an interrupt is eligible only when its priority class is greater than the processor's current priority. Thus, if `CR8 = 8`, classes 0 through 8 are blocked while classes 9 through 15 may be delivered. The APIC's effective Processor Priority Register also accounts for the highest-priority interrupt already in service, so `CR8` is one input to the final decision rather than an interrupt queue by itself. A guest kernel might temporarily raise its priority threshold and later lower it: ```text virtual CR8 = 8 pending virtual interrupt = vector 0x52, class 5 guest executes MOV CR8, 0 -> hardware records virtual CR8 = 0 -> class-5 interrupt becomes eligible -> configured TPR-threshold condition becomes true -> VM exit occurs after the MOV ``` This ordering is exactly what the VMM wants. The `MOV CR8` needs no emulation: the hardware has already updated the virtual task priority. The VMM only needs an opportunity to notice that a pending virtual interrupt may now be delivered. On the exit, saved guest state contains the new priority and guest `RIP` is past the `MOV`; replaying it would be incorrect. TPR shadowing and related APIC-virtualization controls make the common path even cheaper. Reads and writes can update virtual APIC state without exiting, while a threshold exit asks for software only when lowering the priority creates relevant work. Some virtual-APIC and x2APIC operations use the same general design: hardware completes the fast architectural part and exits afterward only if the VMM must finish a routing, notification, or device-model consequence. Current Linux illustrates an important distinction between what the architecture *supports* and how one operating system chooses to use it. During local-APIC initialization, [Linux sets the Task Priority Register to `0x10`](https://codebrowser.dev/linux/linux/arch/x86/kernel/apic/apic.c.html#1544), equivalent to priority class `CR8 = 1`, so APIC vectors `0x10` through `0x1F` cannot be mistaken for processor exceptions. The source notes that Linux does not subsequently change this value. For ordinary critical sections, native x86 Linux primarily [clears and restores `RFLAGS.IF` with `CLI` and `STI`](https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/irqflags.h), which blocks or enables all maskable interrupts rather than dynamically raising and lowering `CR8`. Vector priority still matters to Linux when several interrupts are pending. [Linux reserves high vectors](https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/irq_vectors.h) such as `0xFD` for rescheduling IPIs, `0xFC` and `0xFB` for cross-CPU function calls, and `0xEC` for the local timer. Once interrupts are enabled, the APIC can select the highest-priority eligible pending vector. The high nibble is an APIC delivery priority, however, not a priority attached to every use of an IDT vector: a software `INT 0x80` system call and synchronous processor exceptions are not filtered by `CR8`. #### Example: bus-lock detection A `LOCK`-prefixed instruction does not necessarily lock the memory bus. For an atomic read-modify-write whose operand lies entirely within one cache line in ordinary write-back memory, a modern processor normally obtains exclusive ownership through the cache-coherence protocol: ```text one cacheable line -> request exclusive ownership -> invalidate competing copies of that line -> perform the atomic update -> unrelated cache-line traffic can continue ``` This cache-coherent operation is only *relatively* cheap. A contended atomic variable can still make the line bounce between cores, and false sharing can make unrelated data contend for the same line. Its disturbance is nevertheless concentrated mainly on that cache line. A real bus lock may be required when cache coherence cannot represent the entire atomic operation—for example, when a locked operand crosses a cache-line boundary or targets a memory type for which the ordinary cache-locking mechanism cannot be used. The processor must then serialize access more broadly through the memory interconnect. That can delay other cores and memory agents even when they are accessing unrelated cache lines, making repeated bus locks a possible denial-of-service technique. Bus-lock detection naturally produces a trap-like VM exit because whether the instruction requires a bus lock may depend on information discovered as it executes: - the translated physical address; - the operand's alignment and whether it spans cache lines; - the effective memory type; - the processor's available atomicity mechanism for that access. The processor completes the atomic instruction, reports that it actually caused a bus lock, and then exits. The VMM cannot undo that first expensive transaction, but it can log the event, charge or throttle the vCPU, or terminate a guest that repeats it. The handler must resume after the instruction; replaying it would perform the atomic update—and potentially the bus lock—a second time. VMX bus-lock exiting should not be confused with a separate split-lock-detection policy that can raise an alignment-check exception. The former reports an operation after it caused a bus lock; the latter can be configured as a fault-oriented mechanism for a particular problematic class of locked access. These examples capture the broader rule: > A fault-like exit lets the VMM decide what the operation should do. A trap-like exit lets hardware complete an operation whose result or runtime consequence tells the VMM that software attention is now required. ## Exception priority comes before virtualization policy An instruction can be both architecturally invalid and configured to cause a VM exit. VMX therefore defines priority between ordinary faults and exits. For example: - `RDMSR` at CPL 3 generates `#GP`; it does not give an unprivileged guest program a route to the hypervisor merely because MSR exiting is enabled. - At CPL 0, an intercepted `RDMSR` of an unimplemented index can exit before the processor generates the normal `#GP` for that index. - Operand-fetch faults may occur before an exit whose decision depends on the operand. **Why this makes sense:** Virtualization should preserve the guest’s architectural protection model. Otherwise, configuring an interception could make an instruction behave observably differently even before the hypervisor chose to emulate it. Priority rules define exactly which boundary the VMM inherits. ## Instructions that always exit A small set of instructions cannot execute normally in VMX non-root operation. Chapter 28 includes `CPUID`, `GETSEC`, `INVD`, `XSETBV`, and VMX-management instructions such as `VMXON`, `VMCLEAR`, `VMLAUNCH`, `VMRESUME`, and `VMPTRLD` in the unconditional group, subject to the manual’s detailed fault-priority rules. Guest `VMREAD` and `VMWRITE` are conditional because VMCS shadowing can allow selected field accesses without exits. These instructions fall into several design categories: | Category | Examples | Why unconditional control is useful | |---|---|---| | Machine identity | `CPUID` | The VMM must be able to present a stable virtual CPU model rather than the raw host model | | Global cache or mode behavior | `INVD`, `XSETBV` | Direct execution can affect machine-wide behavior or host-managed extended state | | Secure-launch machinery | `GETSEC` | The guest must not silently alter the platform’s real measured-launch state | | VMX ownership | `VMXON`, `VMLAUNCH`, `VMRESUME` | A guest must not acquire or manipulate the outer hypervisor’s real VMX control structures | | Nested-virtualization interface | `VMREAD`, `VMWRITE`, other VMX instructions | L0 must emulate or selectively accelerate the VMX machine that L1 believes it owns | VMCS shadowing creates an important exception to the simple nested-VMX story: selected guest `VMREAD` and `VMWRITE` operations can access a shadow VMCS without exiting. The shadow still cannot drive real VM entry; L0 retains that authority. ## Conditional exits are the fast-path policy Most interesting instructions are conditional. Their behavior depends on execution controls, bitmaps, masks, or small lookup structures. | Mechanism | Examples | Policy granularity | |---|---|---| | Single control bit | `HLT`, `INVLPG`, `RDPMC`, `RDTSC`, `MOV DR` | All uses of one instruction class | | Dense bitmap | I/O instructions, `RDMSR`, `WRMSR`, `ENCLS`, `PCONFIG` | One port, MSR index, or leaf at a time | | Bit ownership plus shadow | `MOV CR0`, `MOV CR4`, `CLTS`, `LMSW` | Individual control-register bits | | Exact-value targets | `MOV to CR3` | A small set of hot values bypasses a general exit rule | | Timing threshold | `PAUSE` loop exiting, instruction timeout | Exit only after behavior looks costly | | VMREAD/VMWRITE bitmap | Nested-hypervisor VMCS accesses | Individual VMCS field encodings | **Design reading:** These structures mirror the questions hardware must answer on the critical path. “Is this port intercepted?” is a bit lookup. “Does this write cross host-owned CR4 bits?” is a masked comparison. “Is this one of a few permitted CR3 roots?” is a small exact match. VMX avoids a general rule language because predictable hardware lookup is faster and easier to validate. ## Exit controls can also modify the instruction instead of trapping it It is tempting to think that an execution control only chooses between “native” and “VM exit.” Some controls instead give the instruction a virtualized meaning. Examples include: - A masked `MOV to CR0` or `MOV to CR4` may update guest-owned bits while leaving host-owned bits unchanged. - TSC offsetting and scaling transform timestamp reads in hardware. - APIC virtualization changes the behavior of APIC and x2APIC accesses. - `IA32_TSC_DEADLINE` writes can program a guest timer rather than the physical APIC timer. - `IA32_SPEC_CTRL` virtualization can combine guest-visible state with host-enforced bits. - `VMFUNC` can perform an enabled VM function without exiting. This gives VMX three useful paths: ```text ordinary native execution hardware-virtualized execution VM exit for software policy ``` The middle path is how later processor generations remove exits from operations that were previously expensive to virtualize. ## Other events that cause exits Instructions are only one source. Chapter 28 also defines exits for events such as: - exceptions selected by the exception bitmap; - page-fault classes selected by the page-fault mask and match fields; - external interrupts when external-interrupt exiting is enabled; - NMIs when NMI exiting is enabled; - INIT and qualifying SIPI events; - attempted task switches; - expiration of the VMX-preemption timer; - bus locks when VMM bus-lock detection is enabled; - instruction timeout; - interrupt-window and NMI-window conditions. These fall into two broad roles: 1. **Interception:** something happened that the VMM wants to own, such as a selected exception or physical interrupt. 2. **Notification:** a condition the VMM was waiting for has become true, such as “the guest can finally accept an interrupt.” The window-exit controls are especially elegant. They turn a temporal question into a hardware callback. Page faults receive extra filtering because one bitmap bit is often too coarse. Hardware compares the page-fault error code using: ```text (PFEC & PFEC_MASK) == PFEC_MATCH ``` If the comparison matches, exception-bitmap bit 14 has its ordinary meaning; if it does not match, the meaning is reversed. A shadow-paging VMM can therefore intercept the page-fault classes needed to maintain shadow mappings while allowing ordinary guest page faults to reach the guest OS. ## Interrupt-window exiting: wait without single-stepping Suppose a virtual interrupt is pending but the guest has `RFLAGS.IF=0`. The VMM cannot legally inject the interrupt, but exiting after every guest instruction to check `IF` would be disastrous. Instead it enables interrupt-window exiting and resumes the guest: ```text virtual interrupt pending -> guest currently blocks maskable interrupts -> enable interrupt-window exiting -> let guest run -> guest reaches IF=1 with no STI/MOV-SS blocking -> processor exits before the next guest instruction ``` The exit does not represent an error. It is a notification: the requested delivery window is now open. The VMM can disable window exiting, put the virtual interrupt in the VM-entry injection fields, and resume. NMI-window exiting applies the same idea to virtual-NMI blocking. **Why this makes sense:** Hardware already evaluates interruptibility at every boundary. Asking it to notify the VMM when the condition changes is far cheaper than reproducing that observation in software. There is an important distinction between blocking an interrupt *in the guest* and preventing the host from learning about a physical interrupt. When external-interrupt exiting is enabled, guest `RFLAGS.IF` does not prevent an external interrupt from causing a VM exit. A guest that masks its own virtual interrupts must not be able to starve the host’s physical interrupt handling. ## From a physical interrupt to the host handler With external-interrupt exiting enabled, an eligible physical interrupt causes an asynchronous VM exit. No guest instruction caused the exit, so the handler does not advance guest `RIP`; resumption continues at the precise boundary captured by hardware. The processor first saves VMCS-managed guest state and loads the VMCS host state. In particular, it begins at `HOST_RIP` with host `RFLAGS.IF=0`. The entry stub can therefore save the guest general-purpose registers before allowing another maskable interrupt to preempt it. The **acknowledge interrupt on exit** VM-exit control chooses how the physical interrupt reaches host software. “Acknowledge” here means accepting the request from the interrupt controller and obtaining its vector, not completing service of the device. | Control value | What VM exit does | How the host obtains the vector | |---|---|---| | 0 | Leaves the interrupt pending in the local APIC and marks exiting-event information invalid | The VMM eventually enables host interrupts, and the APIC delivers the interrupt normally through the host IDT | | 1 | Acknowledges the local APIC during VM exit, making the request no longer pending, and records the vector in valid exiting-event information | The VM-exit handler reads the vector and explicitly dispatches the corresponding host interrupt path | With acknowledgement enabled, the flow is approximately: ```text physical interrupt arrives while guest runs -> VM exit acknowledges local APIC -> VMCS reports external-interrupt exit and physical vector -> host entry stub saves guest GPRs with IF=0 -> VMM dispatches host handler for that vector -> handler services the device and issues the appropriate APIC EOI -> VMM resumes the guest without advancing guest RIP ``` The APIC acknowledgement does not acknowledge the device itself. A level-triggered device may continue asserting its line until its driver services the source, and the local APIC still needs the normal end-of-interrupt operation. With acknowledgement disabled, the host entry stub instead saves the guest context and later enables host interrupts. The still-pending request then enters through the ordinary host IDT, where the host handler services it and sends EOI before control returns to the VMM. A physical interrupt is also not automatically a guest interrupt. A host NIC might arrive as physical vector `0xE1`; after the host driver or virtual-network backend processes it, the VMM may separately raise the guest virtual NIC’s configured vector, perhaps `0x52`. If the physical device belongs only to the host, no guest interrupt is created. Interrupt remapping, APICv, and posted interrupts can accelerate or bypass parts of this software path for assigned devices. ## NMI exits use host dispatch, not APIC acknowledgement The acknowledge-on-exit control does not apply to NMIs. An NMI has fixed vector 2, is not filtered by `RFLAGS.IF` or `CR8`, and does not use the normal local-APIC in-service and EOI protocol. If **NMI exiting** is enabled, a physical NMI causes a direct VM exit instead of entering the guest’s IDT. Hardware loads `HOST_RIP`; it does not automatically enter host IDT vector 2. Exiting-event information identifies event type NMI and vector 2, after which the VMM must explicitly invoke the host’s NMI-handling machinery: ```text physical NMI -> VM exit to HOST_RIP -> save guest GPRs -> recognize event type NMI, vector 2 -> dispatch host NMI handler -> complete the proper NMI return/unblocking sequence -> VMRESUME ``` For a small bare-metal VMM, the VM-exit routine itself may be the complete NMI handler. A hosted hypervisor such as KVM instead connects the exit to the host operating system’s NMI path so watchdog, profiling, and hardware-diagnostic NMIs retain their normal meaning. A direct NMI exit establishes real NMI blocking after the exit completes, preventing immediate recursive NMIs. The host path must preserve the architecture’s NMI-entry and `IRET`-style unblocking semantics. There is no APIC EOI. A physical host NMI should not ordinarily be injected into the guest; a guest NMI is a separate virtual event created by VMM policy. ## Injecting an interrupt or NMI on VM entry For software-managed delivery, the VMM does not call a guest handler or manually replace guest `RIP`. It writes the VM-entry injected-event identification field and executes `VMLAUNCH` or `VMRESUME`. Hardware loads the guest context and then performs architectural event delivery using the guest IDT or, when enabled, FRED. For an NMI, the basic injected-event encoding is: | Component | Value | |---|---:| | Vector, bits 7:0 | `2` | | Event type, bits 10:8 | `2`, non-maskable interrupt | | Deliver-error-code, bit 11 | `0` | | Valid, bit 31 | `1` | With the remaining reserved bits clear, that is `0x80000202`. On successful entry, hardware consults guest IDT vector 2, performs any required privilege and stack transition, pushes a return frame containing the current guest `RIP`, and begins the guest NMI handler. The VM-entry instruction-length field does not advance `RIP` for an asynchronous NMI. NMI exiting does not intercept an NMI that the VMM explicitly injects on VM entry. Otherwise an injected NMI would immediately exit again and could never reach the guest. A nested exception, invalid guest delivery state, EPT violation while pushing the frame, or another problem encountered *during delivery* can still cause an exit. In that case, original-event information preserves the injected NMI so the VMM can repair the problem without losing or duplicating it. ## STI and MOV-SS interrupt shadows The STI- and MOV-SS-blocking bits are short-lived in terms of **guest architectural progress**, not elapsed time. The CPU tracks them as hidden one-instruction interrupt-shadow state and exposes them in the VMCS so a VM exit can pause the guest between the instruction that created a shadow and the instruction that consumes it. When `STI` changes `RFLAGS.IF` from 0 to 1, `IF` becomes visible as 1 but maskable interrupts are not recognized at the immediately following instruction boundary. The next instruction gets an opportunity to execute first: ```text before STI: IF=0, STI shadow=0 STI completes: IF=1, STI shadow=1 next instruction executes: IF=1, STI shadow=1 next instruction retires: IF=1, STI shadow=0 ``` This makes the classic idle sequence safe: ```asm sti hlt ``` Without the shadow, an interrupt could be handled between `STI` and `HLT`; the CPU could then halt after consuming the only wakeup event. With the shadow, `HLT` executes before the pending interrupt is recognized, and the interrupt wakes the halted processor. `MOV SS` and `POP SS` create a similar one-instruction shadow because older software commonly changed stacks using two instructions: ```asm mov ss, ax mov esp, new_stack_top ``` An interrupt between them would observe the new stack segment with the old stack pointer. The processor therefore sets a hidden latch after successfully loading `SS` and clears it after the following instruction executes. It does not inspect whether the following instruction actually updates the stack pointer. The shadow and associated debug suppression also end if that instruction incurs and delivers an exception. `LSS`, which loads `SS` and the stack pointer together, avoids this two-instruction inconsistency and does not create the same shadow. These latches are governed by retirement and event-delivery boundaries, not by a timer. For example: ```text STI completes and establishes blocking -> intercepted CPUID is the following instruction -> fault-like VM exit occurs before CPUID executes -> saved guest RIP still identifies CPUID -> VMCS saves blocking-by-STI = 1 ``` The host can run for an arbitrary time without expiring the shadow. On resumption, `CPUID` is still the protected next guest instruction. If the next instruction completes before a trap-like exit, the shadow is already cleared in the state saved by that exit. ## What if the VMM injects an NMI while the guest is blocked? The processor does not silently queue an invalid VM-entry injection until the guest becomes ready. VM entry checks the request against guest interruptibility state. In the usual configuration with virtual NMIs enabled, an NMI injection is incompatible with: - blocking by STI; - blocking by MOV SS; - virtual-NMI blocking from an earlier virtual NMI whose handler has not returned. If the VMM nevertheless marks the NMI injection valid and attempts VM entry, no guest instruction and no guest NMI handler executes. The processor reports a VM-entry failure due to invalid guest state, basic reason 33 with the VM-entry-failure indicator set. An attempt to inject an NMI while STI blocking is recorded has the specifically defined exit-qualification value 3. The correct sequence is: ```text mark virtual NMI pending in VMM state -> do not set a valid NMI injection yet -> enable NMI-window exiting -> resume guest -> guest clears STI/MOV-SS or virtual-NMI blocking -> NMI-window VM exit occurs before the next instruction -> disable NMI-window exiting -> program injected NMI, normally 0x80000202 -> resume and let hardware enter guest IDT[2] ``` With **virtual NMIs** enabled, hardware sets virtual-NMI blocking after injecting the NMI, and the guest’s `IRET` clears it. Thus the three VMCS interruptibility bits preserve three different hidden boundaries: | State | Established by | Ends after | |---|---|---| | Blocking by STI | `STI` enabling interrupts | The following instruction executes or incurs an exception | | Blocking by MOV SS | Successful `MOV SS` or `POP SS` | The following instruction executes or incurs an exception | | Virtual-NMI blocking | Delivery of a virtual NMI | Guest `IRET` | ## Direct versus indirect event exits An event can cause a VM exit in two distinct ways. ### Direct exit The event itself is intercepted. For example, the exception bitmap selects `#PF`, or NMI exiting selects an arriving NMI. Because the event has not been delivered to the guest, its normal architectural side effects generally have not occurred. A directly exiting page fault, for example, does not update guest `CR2`; the relevant address is reported through VM-exit information. ### Indirect exit The original event is allowed, but something encountered while trying to deliver it exits. Examples include: - an EPT violation while pushing the guest’s interrupt frame; - a nested exception selected by the exception bitmap; - an APIC-access exit; - a task-switch attempt through an IDT task gate; - a triple fault reached during nested delivery. Here, part of the delivery process may already have happened. The processor records both the new exit cause and the original event being delivered. **Why this distinction matters:** “An interrupt caused the exit” is insufficient. The VMM must know whether the interrupt was never delivered, was partially delivered, or encountered another event. Otherwise it could lose the original event or inject it twice. ## Task switches always leave non-root operation Hardware task switching is not allowed to complete in VMX non-root operation. Attempts cause VM exits after the architecturally higher-priority checks described by Chapter 28. That does not mean the VMM can ignore task-switch semantics. The exit qualification identifies the source of the attempted switch, and faults encountered while resolving task gates or TSS descriptors may take priority. **Design reading:** Hardware task switching mutates a large, memory-resident set of state: TSS images, descriptors, busy bits, `TR`, `RFLAGS.NT`, and potentially privilege state. Making that entire compound transaction independently virtualizable would greatly complicate precise exit boundaries. VMX instead gives the VMM control before the switch commits. ## CET shadow-stack updates expose non-rollbackable boundaries A privilege-changing event or far call may switch supervisor shadow stacks. CET first atomically marks the new stack’s token busy, then may push return information to the new shadow stack. If a later push causes a VM exit, the processor can restore the old active shadow stack, but the memory token in the new stack may remain busy. Intel calls this a **prematurely busy shadow stack**. An optional VM-exit control asks hardware to report that condition in exit information. **Why this field exists:** A transition can cross a point where an architecturally visible memory update cannot simply be rolled back. The VMM needs an explicit diagnostic bit so it can distinguish that partial transition from an ordinary EPT or event-delivery exit and repair or emulate the state correctly. ## The VMX-preemption timer is a non-cooperative scheduling deadline When enabled, VM entry loads a per-vCPU countdown value. It decrements in VMX non-root operation at a capability-reported rate derived from the TSC. Reaching zero causes a VM exit, subject to defined priority and activity-state rules. The timer solves a scheduling problem that guest interrupt masking cannot solve: ```text guest masks interrupts or spins forever -> preemption timer continues -> countdown reaches zero -> VM exit returns control to scheduler ``` It is not the guest’s clock and it is not an APIC timer. It is a VMM-owned bound on how long a vCPU may keep the physical processor. If the “save VMX-preemption timer value” exit control is enabled, an unrelated exit preserves the remaining count in the VMCS. That allows the VMM to resume the same scheduling budget rather than accidentally granting a new full slice. ## Monitor Trap Flag: hypervisor single-stepping without guest TF The monitor trap flag, or MTF, requests a VM exit at a defined instruction boundary after entry. It is useful for hypervisor debugging and for temporarily allowing one guest operation before regaining control. MTF is not just guest `RFLAGS.TF` under another name: | Guest trap flag | Monitor trap flag | |---|---| | Produces guest-visible debug behavior | Produces a VM exit | | Interacts with guest debug state and `#DB` | Belongs to VMM execution policy | | Guest can normally observe or manipulate it | Not exposed as guest architectural state | The detailed rules cover event injection, faults, `REP` iterations, transactional execution, and inactive states because “after one instruction” is not always one simple boundary on x86. **Why this makes sense:** Using guest TF for VMM stepping would contaminate guest debug state and conflict with a guest debugger. MTF provides an orthogonal control plane. ## EPT turns guest physical addresses into an intermediate namespace With Extended Page Tables enabled, guest paging translates a guest linear address to a guest-physical address. EPT then translates that guest-physical address to a machine physical address. ```text guest linear address -> guest page tables guest-physical address -> EPT machine physical address ``` Chapter 28 introduces the runtime role; Chapter 31 specifies the detailed translation mechanism. This second translation gives the guest realistic control over its own “physical” memory while the VMM retains ownership of actual machine pages. An EPT violation means the requested access is inconsistent with the current EPT permissions or mapping. An EPT misconfiguration means the EPT structures themselves contain an invalid encoding; misconfigurations always exit. **Design reading:** Treating guest physical memory as a first-class intermediate address space removes the need to rewrite or shadow every guest page-table change. The guest can manage its own page tables natively while hardware composes them with host ownership. ## VM functions: selected privileged operations without exits `VMFUNC` lets non-root software invoke a function explicitly enabled by the VMM. `EAX` selects the function. The architectural example in Chapter 28 is EPTP switching. For EPTP switching, `ECX` selects an entry from an EPTP list. If the index and selected EPT pointer are valid, hardware switches the active EPT pointer without a VM exit. Potential uses include quickly switching among preconfigured memory views for sandboxing, introspection, or nested components. The permission model is narrow: ```text VMM enables VM functions globally -> VMM enables particular function number -> VMM supplies bounded function-specific data -> guest invokes only that predefined operation ``` **Why this makes sense:** Some operations are safe only within a menu prepared by the VMM. A table-indexed hardware operation provides acceleration without giving the guest arbitrary control over the underlying machine resource. ## Virtualization exceptions move selected EPT handling into the guest Ordinarily an EPT violation exits to the VMM. With the virtualization-exception facility, selected EPT violations can instead become guest exception `#VE` on vector 20. Hardware writes information that would have accompanied the VM exit into a virtualization-exception information area, then delivers `#VE` to the guest if the required conditions hold. This is useful when an enlightened guest component can resolve a virtualization condition locally. It avoids a round trip to the outer VMM. The VMM retains control over convertibility through EPT metadata and VMCS controls. EPT misconfigurations still exit, and violations from certain asynchronous facilities such as Intel PT or PEBS are not converted. **Design reading:** `#VE` is another example of moving policy closer to the consumer. The VMM chooses which conditions are delegable; hardware supplies the same diagnostic facts in a guest-readable channel. ## APIC virtualization removes exits from interrupt bookkeeping Without acceleration, guest accesses to its APIC frequently require exits so the VMM can maintain a virtual interrupt controller. APIC virtualization allows hardware to emulate many APIC accesses, track virtual interrupt state, and deliver virtual interrupts without returning to root operation. This matters because interrupt handling is a high-frequency path: ```text interrupt becomes pending -> priority evaluation -> vector delivery -> in-service tracking -> end-of-interrupt ``` If each arrow exits, the cost can dominate I/O-heavy workloads. Chapter 28 introduces the facility; Chapter 32 supplies the detailed APICv and posted-interrupt machinery. ## Asynchronous producers need address virtualization too Not every memory access is issued directly by a guest instruction. Facilities such as Intel Processor Trace and architectural PEBS can write trace or sampling records asynchronously. VMX controls can make their output addresses guest-physical addresses that are translated through EPT. This prevents a guest from configuring a tracing facility to write directly into arbitrary machine memory. Intel PT adds a subtle buffering problem: an EPT violation discovered only when buffered trace data is finally written could lose data during the resulting exit. Trace-address pre-translation, or TAPT, validates and caches the translation before the buffer must be drained. Any necessary exit therefore occurs early enough for software to repair the mapping without corrupting the trace stream. **Design reading:** DMA-like or trace-producing engines need the same namespace separation as ordinary loads and stores, but their asynchronous buffering creates different restart requirements. Pre-translation moves faults to a recoverable boundary. ## PASID translation virtualizes accelerator identities A **process address-space identifier**, or PASID, is a 20-bit hardware tag that tells a device or accelerator which process address space a request belongs to. It is not a process ID and not a memory address. It is an address-space selector used by devices and the IOMMU. Traditionally, an IOMMU can identify a DMA request using the PCIe device's Requester ID. That works when a device effectively has one owner, but modern accelerators may execute work for many processes simultaneously. Two processes can use the same virtual address for different memory: ```text PASID 17 + address 0x400000 -> process A's page PASID 42 + address 0x400000 -> process B's page ``` The IOMMU uses the device identity, PASID, and address to select the appropriate translation and protection context. In a shared-virtual-addressing configuration, this lets a CPU and an accelerator use the same process virtual addresses. The accelerator can follow pointers in application data structures without software first converting every pointer into a separate device-specific address. PASID resembles several other identifiers, but each belongs to a different layer: | Identifier | What it distinguishes | |---|---| | OS process ID | A software process | | `CR3` | The root of a CPU page-table hierarchy | | PCID | A CPU translation-cache namespace | | PCIe Requester ID | A physical device or function issuing DMA | | PASID | An address-space context within a device | The operating system and IOMMU subsystem allocate and manage PASIDs; applications should not treat them as ordinary process IDs. ### How `ENQCMD` carries a PASID `ENQCMD` and `ENQCMDS` submit 64-byte work descriptors to device portals. The descriptor includes a PASID telling the accelerator which address-space context owns the work. For `ENQCMD`, the PASID normally comes from `IA32_PASID[19:0]`; for the supervisor-oriented `ENQCMDS`, it normally comes from the command data in memory. ```text current process -> PASID -> ENQCMD command submitted to accelerator -> accelerator issues PASID-tagged memory requests -> IOMMU selects that process's mappings and permissions ``` The PASID does not itself translate a memory address. It identifies the context that the IOMMU will use for later address translation and access checks. ### Why a guest PASID cannot be used directly Each guest believes it owns its PASID namespace: ```text VM A: guest PASID 7 VM B: guest PASID 7 ``` The host cannot let both values blindly name the same real IOMMU context. More seriously, allowing a guest to place an arbitrary host PASID in a device command could select an address space belonging to the host or another VM. VMX PASID translation therefore gives each VM a private identifier namespace: ```text VM A guest PASID 7 -> host PASID 1007 VM B guest PASID 7 -> host PASID 2042 ``` For example, the complete path for VM A can be: ```text guest process selects guest PASID 7 -> ENQCMD prepares a 64-byte enqueue store -> VMX replaces guest PASID 7 with host PASID 1007 -> accelerator receives the command tagged with 1007 -> accelerator issues a memory request tagged with 1007 -> IOMMU selects host context 1007 -> access reaches only the permitted host memory ``` This is similar to EPT but translates an identity rather than an address: | Mechanism | Translation performed | |---|---| | EPT | Guest-physical address to host-physical address | | VMX PASID translation | Guest PASID to host PASID | | IOMMU | Device identity plus host PASID plus address to permitted physical memory | ### Why the translation hierarchy has these particular bits VMX uses two VMM-owned PASID-translation hierarchies: ```text guest PASID bit 19 0 -> low PASID directory 1 -> high PASID directory guest PASID bits 18:10 -> one of 512 directory entries guest PASID bits 9:0 -> one of 1024 PASID-table entries selected table entry -> 20-bit host PASID ``` Each root directory and its possible tables cover: ```text 512 directory entries * 1024 table entries = 2^19 PASIDs ``` Bit 19 selects between two such roots, covering the complete 20-bit namespace. This division also keeps the structures sparse: the VMM needs to allocate PASID tables only for ranges a guest actually uses. If a directory entry is absent, a table entry is invalid, or reserved bits are set, PASID translation causes a fault-like VM exit before the enqueue store occurs. The instruction has not submitted work to the device, so the VMM can allocate or validate a host PASID, install the mapping, and safely retry it. **Design reading:** PASID allows one physical device to distinguish memory operations performed for many processes. VMX PASID translation then lets every VM manage a realistic private PASID namespace without exposing the host's real device/IOMMU identities. Performing the replacement in hardware avoids a VM exit for every accelerator submission while enforcing isolation at the point where the identifier would otherwise escape the guest. ## Guest timer: virtual deadline handling without host timer emulation With APIC-timer virtualization, a guest write to `IA32_TSC_DEADLINE` can update a VMX guest deadline rather than the physical APIC timer. The written virtual deadline is interpreted using the active TSC offset and scaling configuration. Hardware converts it into the domain used by the guest timer. When the deadline arrives, hardware: 1. marks the virtual timer vector pending in the virtual APIC; 2. updates virtual-interrupt priority state; 3. evaluates whether the interrupt can be delivered; 4. clears the one-shot guest deadline. **Why this makes sense:** The guest sees a timer defined relative to its virtual TSC. Hardware already knows the time transformation and virtual-APIC state, so it can connect the two without exits on every deadline write and expiration. Do not confuse the two timers: | Timer | Owner | Purpose | |---|---|---| | VMX-preemption timer | Hypervisor | Regain control for scheduling or policy | | Guest timer | Guest-visible virtual APIC | Deliver the guest’s timer interrupt | ## Unrestricted guest: boot states without software contortions Early VMX required `CR0.PE=1` and `CR0.PG=1` in VMX operation. That made real-address mode and unpaged protected mode difficult to represent directly. With the unrestricted-guest control, non-root execution may use `CR0.PE=0` or `CR0.PG=0`. This permits firmware, boot loaders, and early operating-system code to run in their natural modes. EPT must be enabled with unrestricted guest. When guest paging is disabled, the linear address is treated as the guest-physical address and then translated through EPT: ```text CR0.PG = 0 guest linear address = guest-physical address guest-physical address -> EPT -> machine physical address ``` **Why this makes sense:** Relaxing the guest’s mode constraints is safe only if a separate translation layer still protects host memory. EPT provides that invariant. The guest can behave like a reset-era x86 machine without acquiring raw machine-physical access. --- ## Chapter 29: VM entry is a validated transaction `VMLAUNCH` and `VMRESUME` do much more than load `RIP` and change a mode bit. VM entry has to prove that the processor can safely install the requested guest context and, just as importantly, that a future VM exit has a valid host destination. The transition is best understood as a transaction with staged validation: ```text check instruction context -> check controls and host landing state -> check and load guest state -> load entry-list MSRs -> mark launched if appropriate -> perform feature-specific preparation -> inject optional event -> begin guest execution ``` Not every check happens before state loading begins. That fact explains the different failure mechanisms. ## `VMLAUNCH` versus `VMRESUME` The current VMCS has a launch state: - `VMLAUNCH` requires a clear VMCS and is used for its first successful entry after `VMCLEAR`. - `VMRESUME` requires a launched VMCS and is used for later entries. Using the wrong instruction fails rather than guessing the caller’s intent. **Design reading:** First entry and re-entry are different lifecycle events. Separating them lets hardware maintain implementation-private launched state and optimize the common resume path. `VMCLEAR` explicitly returns the object to the pre-launch state and flushes processor-owned VMCS information to memory. Basic checks also reject VM entry while the root context has blocking by `MOV SS`. That blocking describes a transient one-instruction stack-update window in the current host execution stream. It is not guest state and should not leak across the transition. Failing locally keeps the boundary unambiguous rather than carrying a half-completed host stack-update condition into non-root operation. ## The entry stages Chapter 29 gives the high-level order: 1. Perform basic checks on the instruction’s execution context and the current VMCS. 2. Validate VM-execution controls, VM-exit controls, VM-entry controls, and host state. 3. Check guest state while loading it and clearing address-range monitoring. 4. Process the VM-entry MSR-load area. 5. For `VMLAUNCH`, change the VMCS launch state to launched. 6. Perform trace-address pre-translation when required. 7. Inject a configured event or establish a pending MTF exit. This order reflects a safety hierarchy. Before touching guest state, the processor first verifies that it has a valid way back to the host. Once guest loading has begun, failure can no longer be reported as though the VM-entry instruction simply did nothing. ## Four ways entry can fail VM-entry failures are often confused because they do not all return control the same way. | Failure class | Example | Where control goes | Report mechanism | |---|---|---|---| | Ordinary architectural fault | `VMLAUNCH` at the wrong privilege level | Normal host fault handler | `#UD` or `#GP` | | VMfailInvalid | No current VMCS or current VMCS is a shadow VMCS | Instruction following VM entry | `RFLAGS.CF=1` | | VMfailValid | Wrong launch state, invalid controls, bad host state | Instruction following VM entry | `RFLAGS.ZF=1` and VM-instruction error field | | Late VM-entry failure | Invalid guest state or failure loading an entry-list MSR | Host entry point from the VMCS | Exit-reason bit 31 plus reason 33, 34, or 41 | On successful VM entry, control does not fall through to the next host instruction at all. ### Why `CF` and `ZF` encode different failures `CF=1` means the processor could not identify a usable ordinary current VMCS from which to obtain an error field. `ZF=1` means there was a current ordinary VMCS, so the processor can record a VM-instruction error in it. This is the difference between: ```text VMfailInvalid: there is no valid VMCS context for detailed reporting VMfailValid: a VMCS exists, but its lifecycle or contents are invalid ``` The names describe whether the VMX instruction had a valid VMCS reporting context, not whether the entire virtual machine is valid. ### Why late failures look like exits Guest checks and guest-state loading may occur in parallel or in any order. A bad segment field or MSR-list entry may be discovered only after part of the requested guest context has reached the processor. At that point, continuing at the instruction after `VMLAUNCH` with the old host register environment would be unsafe: the old environment may no longer be intact. The processor instead loads the configured host-state area as it would on a VM exit and reports a VM-entry-failure reason. It is exit-like, but not an ordinary exit: - guest state is not saved back to the guest-state area; - the VM-exit MSR-store list is not processed; - most exit-information fields are not updated; - the event-injection valid bit is not cleared; - host state and the VM-exit MSR-load list are used to recover. **Design reading:** This is transactional recovery without pretending full rollback is possible. Early validation failures can return locally. Once the transition has partially committed, recovery uses the already-validated host landing context. ## Why host state is validated before guest state The host-state area describes the recovery environment for every future exit and for late entry failures. The processor checks it before beginning guest loading. Examples of host-state requirements include: - valid fixed-bit settings in host `CR0` and `CR4`; - legal and appropriately aligned `CR3` state; - canonical host `RIP`, `RSP`, and relevant bases on Intel 64 processors; - segment selectors compatible with the normalized host segmentation model; - a valid TR selector and base; - coherent host address-space-size, paging, PAE, PCID, FRED, and EFER relationships; - valid values in any control-gated host fields that will be loaded on exit. The logic is straightforward: > Do not start a guest unless the processor already knows it can construct a usable hypervisor when the guest stops. This makes the exit path dependable even when the guest stops because of an unexpected fault or asynchronous event. ## Control validation is dependency validation VMX controls are not independent feature toggles. Many controls imply other controls, address fields, or capabilities. Examples of relationships include: - unrestricted guest requires EPT; - virtual NMIs requires NMI exiting; - secondary controls are meaningful only when the primary activation bit is enabled; - tertiary controls likewise require their activating primary control; - bitmap and table addresses must be aligned and within the supported physical-address width; - counts for MSR areas must be consistent with address arithmetic and allowed sizes; - an enabled facility makes its associated control-gated fields subject to validation. **Why this makes sense:** A collection of individually legal settings can still describe an impossible machine. Entry validation checks the configuration as a graph of dependencies so hardware never begins execution with half of a feature enabled. ## Guest state must describe one coherent architectural instant The guest-state area contains values that could each look reasonable while being invalid in combination. VM entry checks those relationships. Representative examples include: - `CR0.PG=1` requires `CR0.PE=1`. - An IA-32e guest requires paging and PAE state consistent with long mode. - `CR4.PCIDE` is not valid for a non-IA-32e guest. - `RIP`, SYSENTER pointers, FS/GS bases, and CET-related pointers must satisfy applicable canonical-address rules. - Segment types, privilege levels, present bits, unusable bits, and CS/SS relationships must describe a legal execution context. - Reserved `RFLAGS` bits and mode-dependent bits must have legal values. - Interruptibility bits cannot describe contradictory short-lived states. - Activity state must be supported and compatible with event injection. - PAE PDPTEs must be valid when the selected paging mode requires them. This validation is why a VMCS is not simply a bag of values that hardware blindly copies. **Design reading:** Software restore code normally reaches unusual states through a sequence of architecturally checked instructions. VM entry installs many components at once, bypassing that normal sequence. The processor therefore has to validate the final cross-register invariants explicitly. ## Loading guest state is selective, not a byte-for-byte copy VM entry loads guest control registers, segment caches, descriptor tables, `RIP`, `RSP`, `RFLAGS`, non-register state, and control-gated MSRs. Some bits are fixed, ignored, derived from controls, or normalized. For example: - some reserved or architecturally fixed `CR0` and `DR7` bits are not loaded directly; - EFER long-mode state is tied to the IA-32e guest control and paging state; - FS.base and GS.base correspond to both segment-cache state and architectural MSRs; - the segment unusable bit controls whether other cached attributes matter; - PDPTEs may be loaded from VMCS fields or through the paging structures according to the active mode; - activity and interruptibility fields recreate hidden boundary state. The VMCS describes architectural intent. Entry logic maps that intent to the processor’s valid internal state. ## The VM-entry MSR-load list handles the long tail After ordinary guest-state loading, VM entry processes a list of 16-byte MSR entries: ```text bits 31:0 MSR index bits 63:32 reserved, must be zero bits 127:64 value to load ``` Entries are processed in order. Invalid indexes, reserved bits, disallowed special MSRs, or a value that would fault under the specified rules cause a late VM-entry failure. The ordering matters. A dedicated guest-state MSR field is loaded first; if the same MSR is legally present in the entry MSR-load list, the list value can overwrite it. **Why lists make sense:** Dedicated fields cover frequently used, mode-coupled state that benefits from explicit validation. A count-and-address list supports additional MSRs without expanding the fixed VMCS architecture every time a processor adds one. The list should remain short. Every entry adds transition work, so a VMM normally includes only state that truly differs between host and guest and is safe for this mechanism. ## VM entry clears address-range monitoring VM entry clears monitoring established by `MONITOR` for `MWAIT`-style address monitoring. The monitored range is microarchitectural execution context, not a durable property that should cross from host to guest. Letting a monitor established by host software remain armed in a guest could create cross-context wakeups or information flow. Clearing it provides a clean boundary. VM exit performs the corresponding clear on the way back to the host. ## Event injection happens after the guest context exists If the injected-event field is valid, VM entry delivers the selected interrupt, NMI, exception, software event, `SYSCALL`, or `SYSENTER` after guest state and applicable MSRs have been loaded and VM-execution controls are active. That ordering ensures delivery uses the guest’s environment: - the guest IDTR and IDT, or guest FRED configuration; - guest segment and privilege state; - guest stack or shadow stack; - guest page tables and EPT; - guest interruptibility state; - guest event-delivery controls. For hardware exceptions, NMIs, and external interrupts, the saved return address is the current guest `RIP`. For injected software events, the VM-entry instruction-length field supplies the increment needed to construct the architectural return point. **Why injection is part of entry:** If the VMM first resumed the guest and then tried to interrupt it, at least one unrelated instruction might run. Atomic injection guarantees that the event is the guest’s first architectural action after the transition. ## Injecting an intercepted event does not immediately re-trigger its interception Suppose the exception bitmap intercepts `#PF`. The VMM decides to reinject a page fault on the next entry. The injected page fault itself does not cause a VM exit merely because the `#PF` bitmap bit is set. Otherwise reinjection would loop forever. However, delivery can encounter a new problem. For example, pushing the frame might cause an EPT violation, or the guest IDT descriptor might generate a nested exception selected by the bitmap. That new condition can cause an exit. ```text configured injection -> original event is delivered in guest context -> interception policy does not reject that same injection -> a new nested condition may still exit ``` This distinction preserves both VMM authority and forward progress. ## Original-event information preserves interrupted delivery If injection begins and a nested condition causes an exit, the VMCS records: - the immediate exit cause in the exiting-event and ordinary exit-information fields; - the injected event in the original-event fields. Example: ```text VMM injects guest interrupt vector 50 -> processor begins pushing guest interrupt frame -> stack write causes EPT violation -> VM exit reports EPT violation -> original-event fields preserve vector 50 ``` The VMM can repair the EPT mapping and arrange to complete delivery without inventing or duplicating the interrupt. ## Entry can succeed and still exit before one guest instruction A successful VM entry does not promise that an ordinary guest instruction will execute. Conditions evaluated immediately afterward may cause an exit or event delivery, including: - a preemption timer loaded with zero; - an already-open interrupt window when interrupt-window exiting is set; - an open NMI window; - a TPR-threshold condition; - a pending MTF exit; - a pending debug exception; - a virtual interrupt ready for delivery; - feature-specific preparation such as TAPT encountering a translation problem. This is not a failed entry. The processor constructed the guest context, reached the relevant guest boundary, and then honored the highest-priority pending condition. **Why this matters to debugging:** “No guest instruction retired” does not imply `VMLAUNCH` or `VMRESUME` failed. The exit reason and VM-entry-failure bit distinguish a real entry failure from an immediate post-entry exit. ## Activity and interruptibility state complete the entry context If entry is not injecting an event, the guest activity-state field determines whether the processor becomes active, halted, shutdown, or wait-for-SIPI. If entry injects an event, the processor becomes active because it must deliver that event. Interruptibility fields restore short-lived conditions such as STI blocking, MOV-SS blocking, and NMI or virtual-NMI blocking. Injection modifies some of those rules; for example, successful injection does not leave STI or MOV-SS blocking active afterward. **Design reading:** Entry restores a point on an execution timeline, not merely register contents. Activity and interruptibility determine whether the next thing is an instruction, an event, or no execution at all. --- ## Chapter 30: VM exit is a state transfer plus an explanation A VM exit must do two jobs atomically: 1. Preserve enough guest state to resume or emulate correctly. 2. Establish a host environment in which the VMM can safely execute. It also has to explain the exact boundary it captured. A saved register image without causal information would leave the VMM guessing. The high-level exit sequence is: 1. Record exit information and update relevant entry fields. 2. Save VMCS-managed guest state. 3. Process the VM-exit MSR-store area. 4. Load or synthesize host state and clear address-range monitoring. 5. Process the VM-exit MSR-load area. After that, execution begins at `HOST_RIP` on `HOST_RSP`. Guest general-purpose registers are still in the physical registers until the software trampoline saves them. ## Architectural state is defined before the save begins Chapter 30 first defines what state exists at the instant an exit commences. This is necessary because different causes stop execution at different phases. ### Fault-like instruction exit The instruction did not execute: ```text saved RIP -> intercepted instruction architectural side effects -> absent ``` ### Trap-like instruction exit The operation completed: ```text saved state includes operation's effects saved RIP usually -> following instruction ``` ### Direct event exit The selected event itself exits before guest delivery. Normal delivery side effects generally do not occur. A directly exiting page fault, for example, does not update `CR2`. ### Indirect event exit Delivery began but encountered another exit condition. Some event-related state may already have changed, and incomplete delivery may have written part of a guest stack frame. The processor reports both causal layers. **Why Chapter 30 is so detailed:** Precise virtualization requires the VMM to continue from the same architectural boundary bare hardware would have reached. “Approximately before the event” is not sufficient for nested faults, debugging, NMI blocking, or restartable instructions. ## Exit information is a structured handoff The VMM typically starts with the basic exit reason, then consults reason-specific fields. | Information | Question answered | |---|---| | Exit reason | What broad condition returned control? | | Exit qualification | What reason-specific details apply? | | Guest linear address | Which guest virtual address participated? | | Guest-physical address | Which intermediate physical address participated? | | Exiting-event identification and error code | Which event directly caused the exit? | | Original-event identification and error code | Which event was being delivered when another condition exited? | | Instruction length | How long was the intercepted instruction or software event? | | Instruction information | Which registers, addressing mode, segment, and operand form were decoded? | | VM-instruction error | Why did a VMX instruction fail without an exit? | Not every field is valid for every reason. A handler must dispatch on the exit reason and use only the fields defined for that path. An ordinary VM exit also clears the valid bit in the VM-entry injected-event field. That prevents a successfully consumed injection request from being repeated automatically. If delivery was interrupted, the original-event fields carry the information needed to reconstruct it explicitly. **Design reading:** Variable-format qualification avoids inflating the VMCS with a separate record for every exit class. Hardware reports a common reason plus a compact reason-specific payload. ## Why instruction information saves more than an opcode For selected exits, hardware reports decoded addressing facts such as: - address size and scaling; - segment register; - base and index registers; - whether operands are registers or memory; - operand registers for VMX or descriptor-table instructions. The processor has already decoded this information to execute the instruction. Reporting it avoids a second decoder in the VMM and avoids fetching instruction bytes from memory that may itself be inaccessible or changing. The instruction-length field lets the VMM advance `RIP` after emulating a fault-like instruction: ```text guest RIP = guest RIP + VM-exit instruction length ``` That advance is appropriate only when the handler has completed the instruction’s guest-visible effect. Retrying the instruction instead leaves `RIP` unchanged. ## Guest `RIP` is chosen according to the exit boundary The value saved in the guest `RIP` field is not governed by one universal “current instruction” rule. Representative cases include: - an intercepted fault-like instruction saves the address of that instruction; - an external interrupt or hardware exception saves the return pointer that normal delivery would have used; - interrupt-window and NMI-window exits save the boundary where execution would otherwise continue; - a trap-like exit saves state after the triggering operation; - exits from `HLT` or `MWAIT` inactive states normally preserve the post-instruction continuation point; - software-exception exits retain the instruction boundary needed for correct reinjection semantics. **Why this makes sense:** The saved `RIP` is a restart contract. It tells the VMM whether resumption should retry an operation, continue past it, or deliver an event at the captured boundary. ## VM exit saves VMCS-managed guest state, not the whole vCPU The processor saves fields such as: - `CR0`, `CR3`, and `CR4`; - guest segment caches and descriptor tables; - `RIP`, `RSP`, and `RFLAGS`; - SYSENTER state and control-gated MSRs; - debug state when configured; - activity and interruptibility state; - pending debug state under the defined conditions; - remaining preemption-timer value when requested; - cached PDPTEs when applicable; - virtual-interrupt status and guest deadline when supported. It does not save ordinary general-purpose registers into the VMCS. They remain live in the processor until the host entry trampoline stores them in the software vCPU structure. It also does not automatically switch all x87, SSE, AVX, or other `XSAVE` state. The VMM and host operating system manage that larger state according to their scheduling and ownership policy. The complete vCPU image is therefore: ```text VMCS-managed state + software-saved GPRs + software-managed extended state + virtual-device and scheduler state ``` ## Exit MSR-store and load lists form a context-switch pair After saving ordinary guest state, VM exit can read selected MSRs into the VM-exit MSR-store area. After loading the host-state area, it can load selected host MSRs from the VM-exit MSR-load area. This supports MSRs such as a guest’s `IA32_LSTAR` that do not have dedicated VMCS state fields: ```text guest runs with guest LSTAR -> exit MSR-store saves guest LSTAR -> host-state load establishes basic host context -> exit MSR-load installs host LSTAR ``` The matching VM-entry list can restore guest LSTAR next time. The lists are powerful but dangerous. An invalid entry during VM entry can use the late-failure recovery path. An invalid store or load during VM exit causes a VMX abort because the processor is already committed to leaving the guest and cannot safely report an ordinary guest-visible fault. **Design reading:** Entry has a validated host recovery target, so some entry failures are recoverable. Exit is itself the recovery path. Failure of that path is catastrophic. ## Loading host state creates a landing environment, not the old host snapshot VM exit loads the prepared host-state area: - host `CR0`, `CR3`, and `CR4`; - `HOST_RSP` and `HOST_RIP`; - required segment selectors and bases; - GDTR and IDTR bases; - TR state; - SYSENTER state; - enabled optional MSRs and security state. Other host state is synthesized. Most notably: ```text RFLAGS = 0000000000000002H DR7 = 0000000000000400H IA32_DEBUGCTL = 0 ``` The host begins active, without STI or MOV-SS blocking and without pending debug exceptions. Host segmentation is normalized rather than restored from a complete guest-style descriptor cache. This is why the host-state area is smaller than the guest-state area. The guest can be paused in many awkward architectural states. The VMM chooses one conventional entry point and promises that it satisfies stricter rules. ## Why interrupts begin disabled in the host Because synthesized host `RFLAGS` has `IF=0`, an ordinary maskable interrupt cannot preempt the exit trampoline before it has saved guest registers and established its software frame. The prologue can safely perform something like: ```asm host_rip: push rax push rbx push rcx ; capture remaining guest registers ; establish host ABI state ; optionally enable interrupts later ``` `DF=0` ensures host string operations use the conventional forward direction, and `TF=0` prevents guest or pre-entry single-step state from debugging the host handler. The host’s old flags are not automatically restored because VM exit is not a return to an interrupted host instruction. The VMM’s run wrapper reconstructs any higher-level software continuation it needs. ## The exit handler is a reason-driven state machine A simplified handler looks like: ```text save guest GPRs read basic exit reason switch reason: CPUID: synthesize registers advance guest RIP intercepted MSR: validate/emulate advance RIP or inject #GP EPT violation: inspect qualification and guest-physical address repair mapping, emulate access, inject an event, or stop VM acknowledged external interrupt: read physical vector from exiting-event information dispatch host IRQ, service source, and issue EOI queue a separate virtual interrupt if appropriate unacknowledged external interrupt: preserve guest frame, then allow normal host IDT delivery physical NMI: dispatch host NMI machinery with correct unblocking semantics interrupt window: configure pending virtual interrupt for injection NMI window: configure pending virtual NMI for injection HLT: block vCPU until a wakeup event restore guest GPR policy state VMRESUME ``` The hard part is not the switch statement. It is honoring the restart semantics reported by each exit: which state already changed, which event remains pending, whether `RIP` advances, and whether resumption should inject something first. ## VMX abort: the exit path could not construct a valid host A VMX abort is not an ordinary VM exit and not a guest exception. It means the processor encountered a problem from which the architectural VM-exit mechanism could not safely recover. Examples include: - failure while saving guest MSRs from the exit-store list; - failure loading host MSRs; - invalid host PAE PDPTEs; - corruption that prevents the current VMCS from supporting exit; - an incompatible host address-space configuration; - certain machine-check events during exit. The processor writes a nonzero abort indicator at byte offset 4 of the responsible VMCS region. Outside SMX operation, it then enters a special shutdown state from which only reset recovers. **Why so severe:** An ordinary exception needs a valid execution context in which to run its handler. A VMX abort occurs because the processor cannot reliably finish constructing that context. Continuing with a mixture of guest and host state would be worse than stopping. The abort indicator is memory-resident so another logical processor or post-reset diagnostic code can inspect why the failed CPU stopped. --- ## Three complete round trips The chapter boundaries become clearer when followed end to end. ### Example 1: intercepted `CPUID` ```text 1. Guest executes CPUID. 2. Chapter 28 says CPUID causes a fault-like VM exit. 3. Chapter 30 saves guest RIP pointing at CPUID. 4. Exit reason identifies CPUID; instruction length identifies its size. 5. Host trampoline saves guest GPRs. 6. VMM computes virtual CPUID values into saved RAX/RBX/RCX/RDX. 7. VMM advances guest RIP by the exit instruction length. 8. VMM executes VMRESUME. 9. Chapter 29 validates and reloads the guest. 10. Guest continues after CPUID with the virtual results. ``` The actual instruction never executes with raw host semantics. ### Example 2: pending interrupt while `IF=0` ```text 1. Host queues a virtual interrupt for the vCPU. 2. Guest RFLAGS.IF is zero, so injection would be illegal. 3. VMM enables interrupt-window exiting and resumes. 4. Guest runs until IF=1 and STI/MOV-SS blocking is absent. 5. Chapter 28 causes an interrupt-window VM exit. 6. VMM disables window exiting and configures VM-entry injection. 7. Chapter 29 loads guest state and injects the interrupt atomically. 8. Guest begins in its IDT or FRED handler. ``` The VMM does not poll after every instruction, and the interrupt does not arrive at an illegal boundary. ### Example 3: exit during injected-event delivery ```text 1. VMM configures injection of an external interrupt. 2. VM entry loads the guest and begins event delivery. 3. The processor tries to push the guest interrupt frame. 4. The stack access encounters an EPT violation. 5. VM exit reports the EPT violation as the immediate cause. 6. Original-event fields preserve the injected interrupt. 7. VMM repairs the EPT mapping. 8. VMM arranges retry without losing or duplicating the interrupt. ``` This example is why Chapters 29 and 30 devote so much space to events “incident to” other events. Causal history is part of resumable machine state. --- ## The design patterns behind Chapters 28 through 30 ### Native execution is the default The guest runs directly until a configured boundary is reached. Performance comes from choosing the smallest interception set consistent with isolation and emulation requirements. ### Hardware handles predicates; software handles policy Bitmaps, masks, exact-value targets, priority fields, and time thresholds answer simple fast-path questions. When the answer requires device models, scheduling, security policy, or complex emulation, hardware exits to software. ### Entry validates a future exit before trusting the guest The host landing state is checked first because it is the recovery mechanism for both normal exits and late entry failures. ### Hidden boundary state is first-class state Interrupt shadows, activity state, pending debug events, virtual-NMI blocking, partially completed event delivery, and timer state determine what can happen next even when visible registers match. ### Transition work is capability- and control-gated Optional state is loaded, saved, or validated only when the corresponding facility is supported and enabled. This keeps old hypervisors compatible and avoids paying for unused features. ### The host side is intentionally normalized The guest needs fidelity; the host needs a safe landing pad. Synthesizing host flags and segment-cache properties reduces transition state and makes the first host instruction predictable. ### Failure severity depends on the point of commitment ```text before state loading -> fault or VMfail return during guest construction -> exit-like host recovery during VM exit -> VMX abort ``` The later the failure occurs, the less old context remains available for recovery. --- ## A practical reading checklist When working through a rule in Chapters 28 through 30, ask: 1. Is the operation native, modified, faulting, fault-like exiting, or trap-like exiting? 2. Which control, bitmap, mask, or capability selected that behavior? 3. At what architectural boundary does the transition occur? 4. Has the triggering instruction or event already changed guest state? 5. What does saved guest `RIP` mean for retry versus continuation? 6. Which exit-information fields are defined for this reason? 7. Was another event already being delivered? 8. Does the handler need to advance `RIP`, retry, inject an event, or block the vCPU? 9. Which state is in the VMCS, which remains in registers, and which belongs to software? 10. If entry fails, is it a fault, VMfailInvalid, VMfailValid, or a late exit-like failure? Those questions turn the manual’s many special cases into one consistent task: identify the exact boundary and preserve its semantics. ## The runtime design in one paragraph VMX makes virtualization fast by allowing guest code to use the ordinary execution engine, and safe by evaluating compact hardware policy at the points where guest behavior could affect the machine or require emulation. VM entry is a staged, validated installation of a guest context with a prevalidated host recovery target. VM exit captures the guest at a precisely defined boundary, records enough causal information for software to act correctly, and constructs a normalized host landing environment. The details in Chapters 28 through 30 exist because x86 execution is full of partial operations, nested events, hidden interruptibility, and mode-coupled state; VMX must preserve all of that without turning the common path into software interpretation. --- ## Source and scope This article is an explanatory synthesis of **Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3 (3A, 3B, 3C, & 3D): System Programming Guide**, Order Number **325384-092US**, June 2026. Primary reading: - Chapter 28: VMX non-root operation, exit causes, modified instruction behavior, non-root-specific features, and unrestricted guests - Chapter 29: VM-entry sequencing, validation, state loading, MSR loading, event injection, immediate post-entry behavior, and entry failures - Chapter 30: architectural exit boundaries, exit-information recording, guest-state saving, host-state loading, MSR switching, and VMX aborts Supporting material: - Chapters 26-27: VMX lifecycle, VMCS organization, state fields, and controls - Chapter 31: detailed EPT, VPID, and address-translation behavior - Chapter 32: APIC virtualization, virtual interrupts, and posted interrupts - Appendix A: VMX capability reporting - Appendix C: basic VM-exit reasons The rationale marked as a **design reading** is engineering interpretation of the specified behavior, not a claim that Intel documented the historical or microarchitectural motive. Production hypervisor code must follow the manual’s exact pseudocode, priority rules, capability checks, reserved-bit requirements, errata, and processor-specific documentation.