volt-vmm/docs/firecracker-comparison.md

Firecracker vs Volt: CPU State Setup Comparison

This document compares how Firecracker and Volt set up vCPU state for 64-bit Linux kernel boot.

Executive Summary

Aspect	Firecracker	Volt	Verdict
Boot protocols	PVH + Linux boot	Linux boot (64-bit)	Firecracker more flexible
CR0 flags	Minimal (PE+PG+ET)	Extended (adds WP, NE, AM, MP)	Volt more complete
CR4 flags	Minimal (PAE only)	Extended (adds PGE, OSFXSR, OSXMMEXCPT)	Volt more complete
Page tables	Single identity map (1GB)	Identity + high kernel map	Volt more thorough
Code quality	Battle-tested, production	New implementation	Firecracker proven

---

1. Control Registers

CR0 (Control Register 0)

Bit	Name	Firecracker (Linux)	Volt	Notes
0	PE (Protection Enable)	✅	✅	Required for protected mode
1	MP (Monitor Coprocessor)	❌	✅	FPU monitoring
4	ET (Extension Type)	✅	✅	387 coprocessor present
5	NE (Numeric Error)	❌	✅	Native FPU error handling
16	WP (Write Protect)	❌	✅	Page-level write protection
18	AM (Alignment Mask)	❌	✅	Alignment checking
31	PG (Paging)	✅	✅	Enable paging

Firecracker CR0 values:

// Linux boot:
sregs.cr0 |= X86_CR0_PE;  // After segments/sregs setup
sregs.cr0 |= X86_CR0_PG;  // After page tables setup
// Final: ~0x8000_0001

// PVH boot:
sregs.cr0 = X86_CR0_PE | X86_CR0_ET;  // 0x11
// No paging enabled!

Volt CR0 value:

sregs.cr0 = 0x8003_003B;  // PG | PE | MP | ET | NE | WP | AM

⚠️ Key Difference: Volt enables more CR0 features by default. Firecracker's minimal approach is intentional for PVH (no paging required), but for Linux boot both should work. Volt's WP and NE flags are arguably better defaults for modern kernels.

---

CR3 (Page Table Base)

VMM	Address	Notes
Firecracker	0x9000	PML4 location
Volt	0x1000	PML4 location

Impact: Different page table locations. Both are valid low memory addresses.

---

CR4 (Control Register 4)

Bit	Name	Firecracker	Volt	Notes
5	PAE (Physical Address Extension)	✅	✅	Required for 64-bit
7	PGE (Page Global Enable)	❌	✅	TLB optimization
9	OSFXSR (OS FXSAVE/FXRSTOR)	❌	✅	SSE support
10	OSXMMEXCPT (OS Unmasked SIMD FP)	❌	✅	SIMD exceptions

Firecracker CR4:

sregs.cr4 |= X86_CR4_PAE;  // 0x20
// PVH boot: sregs.cr4 = 0

Volt CR4:

sregs.cr4 = 0x668;  // PAE | PGE | OSFXSR | OSXMMEXCPT

⚠️ Key Difference: Volt enables OSFXSR and OSXMMEXCPT which are required for SSE instructions. Modern Linux kernels expect these. Firecracker relies on the kernel to enable them later.

---

EFER (Extended Feature Enable Register)

Bit	Name	Firecracker (Linux)	Volt	Notes
8	LME (Long Mode Enable)	✅	✅	Enable 64-bit
10	LMA (Long Mode Active)	✅	✅	64-bit active

Both use:

// Firecracker:
sregs.efer |= EFER_LME | EFER_LMA;  // 0x100 | 0x400 = 0x500

// Volt:
sregs.efer = 0x500;  // LME | LMA

✅ Match: Both correctly enable long mode.

---

2. Segment Registers

GDT (Global Descriptor Table)

Firecracker GDT (Linux boot):

// Location: 0x500
[
    gdt_entry(0, 0, 0),            // 0x00: NULL
    gdt_entry(0xa09b, 0, 0xfffff), // 0x08: CODE64 - 64-bit execute/read
    gdt_entry(0xc093, 0, 0xfffff), // 0x10: DATA64 - read/write
    gdt_entry(0x808b, 0, 0xfffff), // 0x18: TSS
]
// Result: CODE64 = 0x00AF_9B00_0000_FFFF
//         DATA64 = 0x00CF_9300_0000_FFFF

Firecracker GDT (PVH boot):

[
    gdt_entry(0, 0, 0),                // 0x00: NULL
    gdt_entry(0xc09b, 0, 0xffff_ffff), // 0x08: CODE32 - 32-bit!
    gdt_entry(0xc093, 0, 0xffff_ffff), // 0x10: DATA
    gdt_entry(0x008b, 0, 0x67),        // 0x18: TSS
]
// Note: 32-bit code segment for PVH protected mode boot

Volt GDT:

// Location: 0x500
CODE64 = 0x00AF_9B00_0000_FFFF  // selector 0x10
DATA64 = 0x00CF_9300_0000_FFFF  // selector 0x18

Segment Selectors

Segment	Firecracker	Volt	Notes
CS	0x08	0x10	Code segment
DS/ES/FS/GS/SS	0x10	0x18	Data segments

⚠️ Key Difference: Firecracker uses GDT entries 1/2 (selectors 0x08/0x10), Volt uses entries 2/3 (selectors 0x10/0x18). Both are valid but could cause issues if assuming specific selector values.

Segment Configuration

Firecracker code segment:

kvm_segment {
    base: 0,
    limit: 0xFFFF_FFFF,  // Scaled from gdt_entry
    selector: 0x08,
    type_: 0xB,          // Execute/Read, accessed
    present: 1,
    dpl: 0,
    db: 0,               // 64-bit mode
    s: 1,
    l: 1,                // Long mode
    g: 1,
}

Volt code segment:

kvm_segment {
    base: 0,
    limit: 0xFFFF_FFFF,
    selector: 0x10,
    type_: 11,           // Execute/Read, accessed
    present: 1,
    dpl: 0,
    db: 0,
    s: 1,
    l: 1,
    g: 1,
}

✅ Match: Segment configurations are functionally identical (just different selectors).

---

3. Page Tables

Memory Layout

Firecracker page tables (Linux boot only):

0x9000: PML4
0xA000: PDPTE
0xB000: PDE (512 × 2MB entries = 1GB coverage)

Volt page tables:

0x1000: PML4
0x2000: PDPT (low memory identity map)
0x3000: PDPT (high kernel 0xFFFFFFFF80000000+)
0x4000+: PD tables (2MB huge pages)

Page Table Entries

Firecracker:

// PML4[0] -> PDPTE
mem.write_obj(boot_pdpte_addr.raw_value() | 0x03, boot_pml4_addr);

// PDPTE[0] -> PDE
mem.write_obj(boot_pde_addr.raw_value() | 0x03, boot_pdpte_addr);

// PDE[i] -> 2MB huge pages
for i in 0..512 {
    mem.write_obj((i << 21) + 0x83u64, boot_pde_addr.unchecked_add(i * 8));
}
// 0x83 = Present | Writable | PageSize (2MB huge page)

Volt:

// PML4[0] -> PDPT_LOW (identity mapping)
let pml4_entry_0 = PDPT_LOW_ADDR | PRESENT | WRITABLE;  // 0x2003

// PML4[511] -> PDPT_HIGH (kernel high mapping)
let pml4_entry_511 = PDPT_HIGH_ADDR | PRESENT | WRITABLE;  // 0x3003

// PD entries use 2MB huge pages
let pd_entry = phys_addr | PRESENT | WRITABLE | PAGE_SIZE;  // 0x83

Coverage

VMM	Identity Map	High Kernel Map
Firecracker	0-1GB	None
Volt	0-4GB	0xFFFFFFFF80000000+ → 0-2GB

⚠️ Key Difference: Volt sets up both identity mapping AND high kernel address mapping (0xFFFFFFFF80000000+). This is more thorough and matches what a real Linux kernel expects. Firecracker only does identity mapping and relies on the kernel to set up its own page tables.

---

4. General Purpose Registers

Initial Register State

Firecracker (Linux boot):

kvm_regs {
    rflags: 0x2,                        // Reserved bit
    rip: entry_point,                   // Kernel entry
    rsp: 0x8ff0,                        // BOOT_STACK_POINTER
    rbp: 0x8ff0,                        // Frame pointer
    rsi: 0x7000,                        // ZERO_PAGE_START (boot_params)
    // All other registers: 0
}

Firecracker (PVH boot):

kvm_regs {
    rflags: 0x2,
    rip: entry_point,
    rbx: 0x6000,                        // PVH_INFO_START
    // All other registers: 0
}

Volt:

kvm_regs {
    rip: kernel_entry,
    rsi: boot_params_addr,              // Linux boot protocol
    rflags: 0x2,
    rsp: 0x8000,                        // Stack pointer
    // All other registers: 0
}

Register	Firecracker (Linux)	Volt	Protocol
RIP	entry_point	kernel_entry	✅
RSI	0x7000	boot_params_addr	Linux boot params
RSP	0x8ff0	0x8000	Stack
RBP	0x8ff0	0	Frame pointer
RFLAGS	0x2	0x2	✅

⚠️ Minor Difference: Firecracker sets RBP to stack pointer, Volt leaves it at 0. Both are valid.

---

5. Memory Layout

Key Addresses

Structure	Firecracker	Volt	Notes
GDT	0x500	0x500	✅ Match
IDT	0x520	0 (limit only)	Volt uses null IDT
Page Tables (PML4)	0x9000	0x1000	Different
PVH start_info	0x6000	0x7000	Different
boot_params/zero_page	0x7000	0x20000	Different
Command line	0x20000	0x8000	Different
E820 map	In zero_page	0x9000	Volt separate
Stack pointer	0x8ff0	0x8000	Different
Kernel load	0x100000 (1MB)	0x100000 (1MB)	✅ Match
TSS address	0xfffbd000	N/A	KVM requirement

E820 Memory Map

Both implementations create similar E820 maps:

Entry 0: 0x0 - 0x9FFFF (640KB) - RAM
Entry 1: 0xA0000 - 0xFFFFF (384KB) - Reserved (legacy hole)
Entry 2: 0x100000 - RAM_END - RAM

---

6. FPU Configuration

Firecracker:

let fpu = kvm_fpu {
    fcw: 0x37f,       // FPU Control Word
    mxcsr: 0x1f80,    // MXCSR - SSE control
    ..Default::default()
};
vcpu.set_fpu(&fpu);

Volt: Currently does not explicitly configure FPU state.

⚠️ Recommendation: Volt should add FPU initialization similar to Firecracker.

---

7. Boot Protocol Support

Protocol	Firecracker	Volt
Linux 64-bit boot	✅	✅
PVH boot	✅	✅ (structures only)
32-bit protected mode entry	✅ (PVH)	❌
EFI handover	❌	❌

Firecracker PVH boot starts in 32-bit protected mode (no paging, CR4=0, CR0=PE|ET), while Volt always starts in 64-bit long mode.

---

8. Recommendations for Volt

High Priority

1. Add FPU initialization:

   let fpu = kvm_fpu {
       fcw: 0x37f,
       mxcsr: 0x1f80,
       ..Default::default()
   };
   self.fd.set_fpu(&fpu)?;
   

2. Consider CR0/CR4 simplification:

   • Your extended flags (WP, NE, AM, PGE, etc.) are fine for modern kernels
   • But may cause issues with older kernels or custom code
   • Firecracker's minimal approach is more universally compatible

Medium Priority

3. Standardize memory layout:

   • Consider aligning with Firecracker's layout for compatibility
   • Especially boot_params at 0x7000 and cmdline at 0x20000

4. Add proper PVH 32-bit boot support:

   • If you want true PVH compatibility, support 32-bit protected mode entry
   • Currently Volt always boots in 64-bit mode

Low Priority

5. Page table coverage:
   • Your dual identity+high mapping is more thorough
   • But Firecracker's 1GB identity map is sufficient for boot
   • Linux kernel sets up its own page tables quickly

---

9. Code References

Firecracker

• src/vmm/src/arch/x86_64/regs.rs - Register setup
• src/vmm/src/arch/x86_64/gdt.rs - GDT construction
• src/vmm/src/arch/x86_64/layout.rs - Memory layout constants
• src/vmm/src/arch/x86_64/mod.rs - Boot configuration

Volt

• vmm/src/kvm/vcpu.rs - vCPU setup (setup_long_mode_with_cr3)
• vmm/src/boot/gdt.rs - GDT setup
• vmm/src/boot/pagetable.rs - Page table setup
• vmm/src/boot/pvh.rs - PVH boot structures
• vmm/src/boot/linux.rs - Linux boot params

---

10. Summary Table

Feature	Firecracker	Volt	Status
CR0	0x80000011	0x8003003B	⚠️ Volt has more flags
CR3	0x9000	0x1000	⚠️ Different
CR4	0x20	0x668	⚠️ Volt has more flags
EFER	0x500	0x500	✅ Match
CS selector	0x08	0x10	⚠️ Different
DS selector	0x10	0x18	⚠️ Different
GDT location	0x500	0x500	✅ Match
Stack pointer	0x8ff0	0x8000	⚠️ Different
boot_params	0x7000	0x20000	⚠️ Different
Kernel load	0x100000	0x100000	✅ Match
FPU init	Yes	No	❌ Missing
PVH 32-bit	Yes	No	❌ Missing
High kernel map	No	Yes	✅ Volt better

---

Document generated: 2026-03-08 Firecracker version: main branch Volt version: current