volt-vmm/docs/firecracker-test-results.md

# Firecracker Kernel Boot Test Results

**Date:** 2026-03-07  
**Firecracker Version:** v1.6.0  
**Test Host:** julius (Linux 6.1.0-42-amd64)

## Executive Summary

**CRITICAL FINDING:** The `vmlinux-5.10` kernel in `kernels/` directory **FAILS TO LOAD** in Firecracker due to corrupted/truncated section headers. The working kernel `vmlinux.bin` (4.14.174) boots successfully in ~93ms.

If Volt is using `vmlinux-5.10`, it will encounter the same ELF loading failure.

---

## Test Results

### Kernel 1: vmlinux-5.10 (FAILS)

**Location:** `projects/volt-vmm/kernels/vmlinux-5.10`  
**Size:** 10.5 MB (10,977,280 bytes)  
**Format:** ELF 64-bit LSB executable, x86-64

**Firecracker Result:**
```
Start microvm error: Cannot load kernel due to invalid memory configuration 
or invalid kernel image: Kernel Loader: failed to load ELF kernel image
```

**Root Cause Analysis:**
```
readelf: Error: Reading 2304 bytes extends past end of file for section headers
```

The ELF file has **missing/corrupted section headers** at offset 43,412,968 (claimed) but file is only 10,977,280 bytes. This is a truncated or improperly built kernel.

---

### Kernel 2: vmlinux.bin (SUCCESS ✓)

**Location:** `comparison/firecracker/vmlinux.bin`  
**Size:** 20.4 MB (21,441,304 bytes)  
**Format:** ELF 64-bit LSB executable, x86-64  
**Version:** Linux 4.14.174

**Boot Result:** SUCCESS  
**Boot Time:** ~93ms to `BOOT_COMPLETE`

**Full Boot Sequence:**
```
[    0.000000] Linux version 4.14.174 (@57edebb99db7) (gcc version 7.5.0)
[    0.000000] Command line: console=ttyS0 reboot=k panic=1 pci=off
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.004000] console [ttyS0] enabled
[    0.032000] smpboot: CPU0: Intel(R) Xeon(R) Processor @ 2.40GHz
[    0.074025] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA. Trying to continue...
[    0.098589] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a U6_16550A
[    0.903994] EXT4-fs (vda): recovery complete
[    0.907903] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    0.916190] Write protecting the kernel read-only data: 12288k
BOOT_COMPLETE 0.93
```

---

## Firecracker Configuration That Works

```json
{
  "boot-source": {
    "kernel_image_path": "./vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "./rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "machine-config": {
    "vcpu_count": 1,
    "mem_size_mib": 128
  }
}
```

**Key boot arguments:**
- `console=ttyS0` - Serial console output
- `reboot=k` - Use keyboard controller for reboot
- `panic=1` - Reboot 1 second after panic
- `pci=off` - Disable PCI (not needed for virtio-mmio)

---

## ELF Structure Comparison

| Property | vmlinux-5.10 (BROKEN) | vmlinux.bin (WORKS) |
|----------|----------------------|---------------------|
| Entry Point | 0x1000000 | 0x1000000 |
| Program Headers | 5 | 5 |
| Section Headers | 36 (claimed) | 36 |
| Section Header Offset | 43,412,968 | 21,439,000 |
| File Size | 10,977,280 | 21,441,304 |
| **Status** | Truncated! | Valid |

The vmlinux-5.10 claims section headers at byte 43MB but file is only 10MB.

---

## Recommendations for Volt

### 1. Use the Working Kernel for Testing
```bash
cp comparison/firecracker/vmlinux.bin kernels/vmlinux-4.14
```

### 2. Rebuild vmlinux-5.10 Properly
If 5.10 is needed, rebuild with:
```bash
make ARCH=x86_64 vmlinux
# Ensure CONFIG_RELOCATABLE=y for Firecracker
# Ensure CONFIG_PHYSICAL_START=0x1000000
```

### 3. Verify Kernel ELF Integrity Before Loading
```bash
readelf -h kernel.bin 2>&1 | grep -q "Error" && echo "CORRUPT"
```

### 4. Critical Kernel Config for VMM
```
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_KVM_GUEST=y
CONFIG_PARAVIRT=y
```

---

## Boot Timeline Analysis (vmlinux.bin)

| Time (ms) | Event |
|-----------|-------|
| 0 | Kernel start, memory setup |
| 4 | Console enabled, TSC calibration |
| 32 | SMP init, CPU brought up |
| 74 | virtio-mmio device registered |
| 99 | Serial driver loaded (ttyS0) |
| 385 | i8042 keyboard init |
| 897 | Root filesystem mounted |
| 920 | Kernel read-only protection |
| 930 | BOOT_COMPLETE |

**Total boot time: ~93ms to userspace**

---

## Commands Used

```bash
# Start Firecracker with API socket
./firecracker --api-sock /tmp/fc.sock &

# Configure boot source
curl -s --unix-socket /tmp/fc.sock -X PUT "http://localhost/boot-source" \
  -H "Content-Type: application/json" \
  -d '{"kernel_image_path": "./vmlinux.bin", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"}'

# Configure rootfs
curl -s --unix-socket /tmp/fc.sock -X PUT "http://localhost/drives/rootfs" \
  -H "Content-Type: application/json" \
  -d '{"drive_id": "rootfs", "path_on_host": "./rootfs.ext4", "is_root_device": true, "is_read_only": false}'

# Configure machine
curl -s --unix-socket /tmp/fc.sock -X PUT "http://localhost/machine-config" \
  -H "Content-Type: application/json" \
  -d '{"vcpu_count": 1, "mem_size_mib": 128}'

# Start VM
curl -s --unix-socket /tmp/fc.sock -X PUT "http://localhost/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "InstanceStart"}'
```

---

## Conclusion

The kernel issue is **not with Firecracker or Volt's VMM** - it's a corrupted kernel image. The `vmlinux.bin` kernel (4.14.174) proves that Firecracker can successfully boot VMs on this host with proper kernel images.

**Action Required:** Use `vmlinux.bin` for Volt testing, or rebuild `vmlinux-5.10` from source with complete ELF sections.