Volt VMM (Neutron Stardust): source-available under AGPSL v5.0
KVM-based microVMM for the Volt platform: - Sub-second VM boot times - Minimal memory footprint - Landlock LSM + seccomp security - Virtio device support - Custom kernel management Copyright (c) Armored Gates LLC. All rights reserved. Licensed under AGPSL v5.0
This commit is contained in:
Karl Clinger
434
docs/firecracker-comparison.md
Normal file
434
docs/firecracker-comparison.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# Firecracker vs Volt: CPU State Setup Comparison
|
||||
|
||||
This document compares how Firecracker and Volt set up vCPU state for 64-bit Linux kernel boot.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| Aspect | Firecracker | Volt | Verdict |
|
||||
|--------|-------------|-----------|---------|
|
||||
| Boot protocols | PVH + Linux boot | Linux boot (64-bit) | Firecracker more flexible |
|
||||
| CR0 flags | Minimal (PE+PG+ET) | Extended (adds WP, NE, AM, MP) | Volt more complete |
|
||||
| CR4 flags | Minimal (PAE only) | Extended (adds PGE, OSFXSR, OSXMMEXCPT) | Volt more complete |
|
||||
| Page tables | Single identity map (1GB) | Identity + high kernel map | Volt more thorough |
|
||||
| Code quality | Battle-tested, production | New implementation | Firecracker proven |
|
||||
|
||||
---
|
||||
|
||||
## 1. Control Registers
|
||||
|
||||
### CR0 (Control Register 0)
|
||||
|
||||
| Bit | Name | Firecracker (Linux) | Volt | Notes |
|
||||
|-----|------|---------------------|-----------|-------|
|
||||
| 0 | PE (Protection Enable) | ✅ | ✅ | Required for protected mode |
|
||||
| 1 | MP (Monitor Coprocessor) | ❌ | ✅ | FPU monitoring |
|
||||
| 4 | ET (Extension Type) | ✅ | ✅ | 387 coprocessor present |
|
||||
| 5 | NE (Numeric Error) | ❌ | ✅ | Native FPU error handling |
|
||||
| 16 | WP (Write Protect) | ❌ | ✅ | Page-level write protection |
|
||||
| 18 | AM (Alignment Mask) | ❌ | ✅ | Alignment checking |
|
||||
| 31 | PG (Paging) | ✅ | ✅ | Enable paging |
|
||||
|
||||
**Firecracker CR0 values:**
|
||||
```rust
|
||||
// Linux boot:
|
||||
sregs.cr0 |= X86_CR0_PE; // After segments/sregs setup
|
||||
sregs.cr0 |= X86_CR0_PG; // After page tables setup
|
||||
// Final: ~0x8000_0001
|
||||
|
||||
// PVH boot:
|
||||
sregs.cr0 = X86_CR0_PE | X86_CR0_ET; // 0x11
|
||||
// No paging enabled!
|
||||
```
|
||||
|
||||
**Volt CR0 value:**
|
||||
```rust
|
||||
sregs.cr0 = 0x8003_003B; // PG | PE | MP | ET | NE | WP | AM
|
||||
```
|
||||
|
||||
**⚠️ Key Difference:** Volt enables more CR0 features by default. Firecracker's minimal approach is intentional for PVH (no paging required), but for Linux boot both should work. Volt's WP and NE flags are arguably better defaults for modern kernels.
|
||||
|
||||
---
|
||||
|
||||
### CR3 (Page Table Base)
|
||||
|
||||
| VMM | Address | Notes |
|
||||
|-----|---------|-------|
|
||||
| Firecracker | `0x9000` | PML4 location |
|
||||
| Volt | `0x1000` | PML4 location |
|
||||
|
||||
**Impact:** Different page table locations. Both are valid low memory addresses.
|
||||
|
||||
---
|
||||
|
||||
### CR4 (Control Register 4)
|
||||
|
||||
| Bit | Name | Firecracker | Volt | Notes |
|
||||
|-----|------|-------------|-----------|-------|
|
||||
| 5 | PAE (Physical Address Extension) | ✅ | ✅ | Required for 64-bit |
|
||||
| 7 | PGE (Page Global Enable) | ❌ | ✅ | TLB optimization |
|
||||
| 9 | OSFXSR (OS FXSAVE/FXRSTOR) | ❌ | ✅ | SSE support |
|
||||
| 10 | OSXMMEXCPT (OS Unmasked SIMD FP) | ❌ | ✅ | SIMD exceptions |
|
||||
|
||||
**Firecracker CR4:**
|
||||
```rust
|
||||
sregs.cr4 |= X86_CR4_PAE; // 0x20
|
||||
// PVH boot: sregs.cr4 = 0
|
||||
```
|
||||
|
||||
**Volt CR4:**
|
||||
```rust
|
||||
sregs.cr4 = 0x668; // PAE | PGE | OSFXSR | OSXMMEXCPT
|
||||
```
|
||||
|
||||
**⚠️ Key Difference:** Volt enables OSFXSR and OSXMMEXCPT which are required for SSE instructions. Modern Linux kernels expect these. Firecracker relies on the kernel to enable them later.
|
||||
|
||||
---
|
||||
|
||||
### EFER (Extended Feature Enable Register)
|
||||
|
||||
| Bit | Name | Firecracker (Linux) | Volt | Notes |
|
||||
|-----|------|---------------------|-----------|-------|
|
||||
| 8 | LME (Long Mode Enable) | ✅ | ✅ | Enable 64-bit |
|
||||
| 10 | LMA (Long Mode Active) | ✅ | ✅ | 64-bit active |
|
||||
|
||||
**Both use:**
|
||||
```rust
|
||||
// Firecracker:
|
||||
sregs.efer |= EFER_LME | EFER_LMA; // 0x100 | 0x400 = 0x500
|
||||
|
||||
// Volt:
|
||||
sregs.efer = 0x500; // LME | LMA
|
||||
```
|
||||
|
||||
**✅ Match:** Both correctly enable long mode.
|
||||
|
||||
---
|
||||
|
||||
## 2. Segment Registers
|
||||
|
||||
### GDT (Global Descriptor Table)
|
||||
|
||||
**Firecracker GDT (Linux boot):**
|
||||
```rust
|
||||
// Location: 0x500
|
||||
[
|
||||
gdt_entry(0, 0, 0), // 0x00: NULL
|
||||
gdt_entry(0xa09b, 0, 0xfffff), // 0x08: CODE64 - 64-bit execute/read
|
||||
gdt_entry(0xc093, 0, 0xfffff), // 0x10: DATA64 - read/write
|
||||
gdt_entry(0x808b, 0, 0xfffff), // 0x18: TSS
|
||||
]
|
||||
// Result: CODE64 = 0x00AF_9B00_0000_FFFF
|
||||
// DATA64 = 0x00CF_9300_0000_FFFF
|
||||
```
|
||||
|
||||
**Firecracker GDT (PVH boot):**
|
||||
```rust
|
||||
[
|
||||
gdt_entry(0, 0, 0), // 0x00: NULL
|
||||
gdt_entry(0xc09b, 0, 0xffff_ffff), // 0x08: CODE32 - 32-bit!
|
||||
gdt_entry(0xc093, 0, 0xffff_ffff), // 0x10: DATA
|
||||
gdt_entry(0x008b, 0, 0x67), // 0x18: TSS
|
||||
]
|
||||
// Note: 32-bit code segment for PVH protected mode boot
|
||||
```
|
||||
|
||||
**Volt GDT:**
|
||||
```rust
|
||||
// Location: 0x500
|
||||
CODE64 = 0x00AF_9B00_0000_FFFF // selector 0x10
|
||||
DATA64 = 0x00CF_9300_0000_FFFF // selector 0x18
|
||||
```
|
||||
|
||||
### Segment Selectors
|
||||
|
||||
| Segment | Firecracker | Volt | Notes |
|
||||
|---------|-------------|-----------|-------|
|
||||
| CS | 0x08 | 0x10 | Code segment |
|
||||
| DS/ES/FS/GS/SS | 0x10 | 0x18 | Data segments |
|
||||
|
||||
**⚠️ Key Difference:** Firecracker uses GDT entries 1/2 (selectors 0x08/0x10), Volt uses entries 2/3 (selectors 0x10/0x18). Both are valid but could cause issues if assuming specific selector values.
|
||||
|
||||
### Segment Configuration
|
||||
|
||||
**Firecracker code segment:**
|
||||
```rust
|
||||
kvm_segment {
|
||||
base: 0,
|
||||
limit: 0xFFFF_FFFF, // Scaled from gdt_entry
|
||||
selector: 0x08,
|
||||
type_: 0xB, // Execute/Read, accessed
|
||||
present: 1,
|
||||
dpl: 0,
|
||||
db: 0, // 64-bit mode
|
||||
s: 1,
|
||||
l: 1, // Long mode
|
||||
g: 1,
|
||||
}
|
||||
```
|
||||
|
||||
**Volt code segment:**
|
||||
```rust
|
||||
kvm_segment {
|
||||
base: 0,
|
||||
limit: 0xFFFF_FFFF,
|
||||
selector: 0x10,
|
||||
type_: 11, // Execute/Read, accessed
|
||||
present: 1,
|
||||
dpl: 0,
|
||||
db: 0,
|
||||
s: 1,
|
||||
l: 1,
|
||||
g: 1,
|
||||
}
|
||||
```
|
||||
|
||||
**✅ Match:** Segment configurations are functionally identical (just different selectors).
|
||||
|
||||
---
|
||||
|
||||
## 3. Page Tables
|
||||
|
||||
### Memory Layout
|
||||
|
||||
**Firecracker page tables (Linux boot only):**
|
||||
```
|
||||
0x9000: PML4
|
||||
0xA000: PDPTE
|
||||
0xB000: PDE (512 × 2MB entries = 1GB coverage)
|
||||
```
|
||||
|
||||
**Volt page tables:**
|
||||
```
|
||||
0x1000: PML4
|
||||
0x2000: PDPT (low memory identity map)
|
||||
0x3000: PDPT (high kernel 0xFFFFFFFF80000000+)
|
||||
0x4000+: PD tables (2MB huge pages)
|
||||
```
|
||||
|
||||
### Page Table Entries
|
||||
|
||||
**Firecracker:**
|
||||
```rust
|
||||
// PML4[0] -> PDPTE
|
||||
mem.write_obj(boot_pdpte_addr.raw_value() | 0x03, boot_pml4_addr);
|
||||
|
||||
// PDPTE[0] -> PDE
|
||||
mem.write_obj(boot_pde_addr.raw_value() | 0x03, boot_pdpte_addr);
|
||||
|
||||
// PDE[i] -> 2MB huge pages
|
||||
for i in 0..512 {
|
||||
mem.write_obj((i << 21) + 0x83u64, boot_pde_addr.unchecked_add(i * 8));
|
||||
}
|
||||
// 0x83 = Present | Writable | PageSize (2MB huge page)
|
||||
```
|
||||
|
||||
**Volt:**
|
||||
```rust
|
||||
// PML4[0] -> PDPT_LOW (identity mapping)
|
||||
let pml4_entry_0 = PDPT_LOW_ADDR | PRESENT | WRITABLE; // 0x2003
|
||||
|
||||
// PML4[511] -> PDPT_HIGH (kernel high mapping)
|
||||
let pml4_entry_511 = PDPT_HIGH_ADDR | PRESENT | WRITABLE; // 0x3003
|
||||
|
||||
// PD entries use 2MB huge pages
|
||||
let pd_entry = phys_addr | PRESENT | WRITABLE | PAGE_SIZE; // 0x83
|
||||
```
|
||||
|
||||
### Coverage
|
||||
|
||||
| VMM | Identity Map | High Kernel Map |
|
||||
|-----|--------------|-----------------|
|
||||
| Firecracker | 0-1GB | None |
|
||||
| Volt | 0-4GB | 0xFFFFFFFF80000000+ → 0-2GB |
|
||||
|
||||
**⚠️ Key Difference:** Volt sets up both identity mapping AND high kernel address mapping (0xFFFFFFFF80000000+). This is more thorough and matches what a real Linux kernel expects. Firecracker only does identity mapping and relies on the kernel to set up its own page tables.
|
||||
|
||||
---
|
||||
|
||||
## 4. General Purpose Registers
|
||||
|
||||
### Initial Register State
|
||||
|
||||
**Firecracker (Linux boot):**
|
||||
```rust
|
||||
kvm_regs {
|
||||
rflags: 0x2, // Reserved bit
|
||||
rip: entry_point, // Kernel entry
|
||||
rsp: 0x8ff0, // BOOT_STACK_POINTER
|
||||
rbp: 0x8ff0, // Frame pointer
|
||||
rsi: 0x7000, // ZERO_PAGE_START (boot_params)
|
||||
// All other registers: 0
|
||||
}
|
||||
```
|
||||
|
||||
**Firecracker (PVH boot):**
|
||||
```rust
|
||||
kvm_regs {
|
||||
rflags: 0x2,
|
||||
rip: entry_point,
|
||||
rbx: 0x6000, // PVH_INFO_START
|
||||
// All other registers: 0
|
||||
}
|
||||
```
|
||||
|
||||
**Volt:**
|
||||
```rust
|
||||
kvm_regs {
|
||||
rip: kernel_entry,
|
||||
rsi: boot_params_addr, // Linux boot protocol
|
||||
rflags: 0x2,
|
||||
rsp: 0x8000, // Stack pointer
|
||||
// All other registers: 0
|
||||
}
|
||||
```
|
||||
|
||||
| Register | Firecracker (Linux) | Volt | Protocol |
|
||||
|----------|---------------------|-----------|----------|
|
||||
| RIP | entry_point | kernel_entry | ✅ |
|
||||
| RSI | 0x7000 | boot_params_addr | Linux boot params |
|
||||
| RSP | 0x8ff0 | 0x8000 | Stack |
|
||||
| RBP | 0x8ff0 | 0 | Frame pointer |
|
||||
| RFLAGS | 0x2 | 0x2 | ✅ |
|
||||
|
||||
**⚠️ Minor Difference:** Firecracker sets RBP to stack pointer, Volt leaves it at 0. Both are valid.
|
||||
|
||||
---
|
||||
|
||||
## 5. Memory Layout
|
||||
|
||||
### Key Addresses
|
||||
|
||||
| Structure | Firecracker | Volt | Notes |
|
||||
|-----------|-------------|-----------|-------|
|
||||
| GDT | 0x500 | 0x500 | ✅ Match |
|
||||
| IDT | 0x520 | 0 (limit only) | Volt uses null IDT |
|
||||
| Page Tables (PML4) | 0x9000 | 0x1000 | Different |
|
||||
| PVH start_info | 0x6000 | 0x7000 | Different |
|
||||
| boot_params/zero_page | 0x7000 | 0x20000 | Different |
|
||||
| Command line | 0x20000 | 0x8000 | Different |
|
||||
| E820 map | In zero_page | 0x9000 | Volt separate |
|
||||
| Stack pointer | 0x8ff0 | 0x8000 | Different |
|
||||
| Kernel load | 0x100000 (1MB) | 0x100000 (1MB) | ✅ Match |
|
||||
| TSS address | 0xfffbd000 | N/A | KVM requirement |
|
||||
|
||||
### E820 Memory Map
|
||||
|
||||
Both implementations create similar E820 maps:
|
||||
|
||||
```
|
||||
Entry 0: 0x0 - 0x9FFFF (640KB) - RAM
|
||||
Entry 1: 0xA0000 - 0xFFFFF (384KB) - Reserved (legacy hole)
|
||||
Entry 2: 0x100000 - RAM_END - RAM
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. FPU Configuration
|
||||
|
||||
**Firecracker:**
|
||||
```rust
|
||||
let fpu = kvm_fpu {
|
||||
fcw: 0x37f, // FPU Control Word
|
||||
mxcsr: 0x1f80, // MXCSR - SSE control
|
||||
..Default::default()
|
||||
};
|
||||
vcpu.set_fpu(&fpu);
|
||||
```
|
||||
|
||||
**Volt:** Currently does not explicitly configure FPU state.
|
||||
|
||||
**⚠️ Recommendation:** Volt should add FPU initialization similar to Firecracker.
|
||||
|
||||
---
|
||||
|
||||
## 7. Boot Protocol Support
|
||||
|
||||
| Protocol | Firecracker | Volt |
|
||||
|----------|-------------|-----------|
|
||||
| Linux 64-bit boot | ✅ | ✅ |
|
||||
| PVH boot | ✅ | ✅ (structures only) |
|
||||
| 32-bit protected mode entry | ✅ (PVH) | ❌ |
|
||||
| EFI handover | ❌ | ❌ |
|
||||
|
||||
**Firecracker PVH boot** starts in 32-bit protected mode (no paging, CR4=0, CR0=PE|ET), while **Volt** always starts in 64-bit long mode.
|
||||
|
||||
---
|
||||
|
||||
## 8. Recommendations for Volt
|
||||
|
||||
### High Priority
|
||||
|
||||
1. **Add FPU initialization:**
|
||||
```rust
|
||||
let fpu = kvm_fpu {
|
||||
fcw: 0x37f,
|
||||
mxcsr: 0x1f80,
|
||||
..Default::default()
|
||||
};
|
||||
self.fd.set_fpu(&fpu)?;
|
||||
```
|
||||
|
||||
2. **Consider CR0/CR4 simplification:**
|
||||
- Your extended flags (WP, NE, AM, PGE, etc.) are fine for modern kernels
|
||||
- But may cause issues with older kernels or custom code
|
||||
- Firecracker's minimal approach is more universally compatible
|
||||
|
||||
### Medium Priority
|
||||
|
||||
3. **Standardize memory layout:**
|
||||
- Consider aligning with Firecracker's layout for compatibility
|
||||
- Especially boot_params at 0x7000 and cmdline at 0x20000
|
||||
|
||||
4. **Add proper PVH 32-bit boot support:**
|
||||
- If you want true PVH compatibility, support 32-bit protected mode entry
|
||||
- Currently Volt always boots in 64-bit mode
|
||||
|
||||
### Low Priority
|
||||
|
||||
5. **Page table coverage:**
|
||||
- Your dual identity+high mapping is more thorough
|
||||
- But Firecracker's 1GB identity map is sufficient for boot
|
||||
- Linux kernel sets up its own page tables quickly
|
||||
|
||||
---
|
||||
|
||||
## 9. Code References
|
||||
|
||||
### Firecracker
|
||||
- `src/vmm/src/arch/x86_64/regs.rs` - Register setup
|
||||
- `src/vmm/src/arch/x86_64/gdt.rs` - GDT construction
|
||||
- `src/vmm/src/arch/x86_64/layout.rs` - Memory layout constants
|
||||
- `src/vmm/src/arch/x86_64/mod.rs` - Boot configuration
|
||||
|
||||
### Volt
|
||||
- `vmm/src/kvm/vcpu.rs` - vCPU setup (`setup_long_mode_with_cr3`)
|
||||
- `vmm/src/boot/gdt.rs` - GDT setup
|
||||
- `vmm/src/boot/pagetable.rs` - Page table setup
|
||||
- `vmm/src/boot/pvh.rs` - PVH boot structures
|
||||
- `vmm/src/boot/linux.rs` - Linux boot params
|
||||
|
||||
---
|
||||
|
||||
## 10. Summary Table
|
||||
|
||||
| Feature | Firecracker | Volt | Status |
|
||||
|---------|-------------|-----------|--------|
|
||||
| CR0 | 0x80000011 | 0x8003003B | ⚠️ Volt has more flags |
|
||||
| CR3 | 0x9000 | 0x1000 | ⚠️ Different |
|
||||
| CR4 | 0x20 | 0x668 | ⚠️ Volt has more flags |
|
||||
| EFER | 0x500 | 0x500 | ✅ Match |
|
||||
| CS selector | 0x08 | 0x10 | ⚠️ Different |
|
||||
| DS selector | 0x10 | 0x18 | ⚠️ Different |
|
||||
| GDT location | 0x500 | 0x500 | ✅ Match |
|
||||
| Stack pointer | 0x8ff0 | 0x8000 | ⚠️ Different |
|
||||
| boot_params | 0x7000 | 0x20000 | ⚠️ Different |
|
||||
| Kernel load | 0x100000 | 0x100000 | ✅ Match |
|
||||
| FPU init | Yes | No | ❌ Missing |
|
||||
| PVH 32-bit | Yes | No | ❌ Missing |
|
||||
| High kernel map | No | Yes | ✅ Volt better |
|
||||
|
||||
---
|
||||
|
||||
*Document generated: 2026-03-08*
|
||||
*Firecracker version: main branch*
|
||||
*Volt version: current*
|
||||
Reference in New Issue
Block a user