volt-vmm/docs/MEMORY_LAYOUT_ANALYSIS.md

Volt ELF Loading & Memory Layout Analysis

Date: 2025-01-20
Status: ✅ ALL ISSUES RESOLVED
Kernel: vmlinux with Virtual 0xffffffff81000000 → Physical 0x1000000, Entry at physical 0x1000000

Executive Summary

Component	Status	Notes
ELF Loading	✅ Correct	Loads to correct physical addresses
Entry Point	✅ Correct	Virtual address used (page tables handle translation)
RSI → boot_params	✅ Correct	RSI set to BOOT_PARAMS_ADDR (0x20000)
Page Tables (identity)	✅ Correct	Maps physical 0-4GB to virtual 0-4GB
Page Tables (high-half)	✅ Correct	Maps 0xffffffff80000000+ to physical 0+
Memory Layout	✅ FIXED	Addresses relocated above page table area
Constants	✅ FIXED	Cleaned up and documented

---

1. ELF Loading Analysis (loader.rs)

Current Implementation

let dest_addr = if ph.p_paddr >= layout::HIGH_MEMORY_START {
    ph.p_paddr
} else {
    load_addr + ph.p_paddr
};

Verification

For vmlinux with:

• p_paddr = 0x1000000 (16MB physical)
• p_vaddr = 0xffffffff81000000 (high-half virtual)

The code correctly:

1. Detects p_paddr (0x1000000) >= HIGH_MEMORY_START (0x100000) → true
2. Uses p_paddr directly as dest_addr = 0x1000000
3. Loads kernel to physical address 0x1000000 ✅

Entry Point

entry_point: elf.e_entry,  // Returns virtual address (e.g., 0xffffffff81000000 + startup_64_offset)

This is correct because the page tables map the virtual address to the correct physical location.

---

2. Memory Layout Analysis

Current Memory Map

Physical Address    Size      Structure
─────────────────────────────────────────
0x0000 - 0x04FF    0x500     Reserved (IVT, BDA)
0x0500 - 0x052F    0x030     GDT (3 entries)
0x0530 - 0x0FFF    ~0xAD0    Unused gap
0x1000 - 0x1FFF    0x1000    PML4 (Page Map Level 4)
0x2000 - 0x2FFF    0x1000    PDPT_LOW (identity mapping)
0x3000 - 0x3FFF    0x1000    PDPT_HIGH (kernel mapping)
0x4000 - 0x7FFF    0x4000    PD tables (for identity mapping, up to 4GB)
                              ├─ 0x4000: PD for 0-1GB
                              ├─ 0x5000: PD for 1-2GB
                              ├─ 0x6000: PD for 2-3GB
                              └─ 0x7000: PD for 3-4GB  ← OVERLAP!
0x7000 - 0x7FFF    0x1000    boot_params (Linux zero page) ← COLLISION!
0x8000 - 0x8FFF    0x1000    CMDLINE
0x8000+            0x2000    PD tables for high-half kernel mapping
0x9000 - 0x9XXX    ~0x500    E820 memory map
...
0x100000           varies    Kernel load address (1MB)
0x1000000          varies    Kernel (16MB physical for vmlinux)

🔴 CRITICAL: Memory Overlap

Problem: For guest memory sizes > 512MB, the page directory tables for identity mapping extend into 0x7000, which is also used for boot_params.

Memory Size    PD Tables Needed    PD Address Range    Overlaps boot_params?
─────────────────────────────────────────────────────────────────────────────
128 MB         1                   0x4000-0x4FFF       No
512 MB         1                   0x4000-0x4FFF       No
1 GB           1                   0x4000-0x4FFF       No
2 GB           2                   0x4000-0x5FFF       No
3 GB           2                   0x4000-0x5FFF       No
4 GB           2                   0x4000-0x5FFF       No (but close)

Wait - rechecking the math:

• Each PD covers 1GB (512 entries × 2MB per entry)
• For 4GB identity mapping: need ceil(4GB / 1GB) = 4 PD tables

Actually looking at the code again:

let num_2mb_pages = (map_size + 0x1FFFFF) / 0x200000;
let num_pd_tables = ((num_2mb_pages + 511) / 512).max(1) as usize;

For 4GB = 4 * 1024 * 1024 * 1024 bytes:

• num_2mb_pages = 4GB / 2MB = 2048 pages
• num_pd_tables = (2048 + 511) / 512 = 4 (capped at 4 by .min(4) in the loop)

The 4 PD tables are at 0x4000, 0x5000, 0x6000, 0x7000 - overlapping boot_params!

Then high_pd_base:

let high_pd_base = PD_ADDR + (num_pd_tables.min(4) as u64 * PAGE_TABLE_SIZE);

= 0x4000 + 4 * 0x1000 = 0x8000 - overlapping CMDLINE!

---

3. Page Table Mapping Verification

High-Half Kernel Mapping (0xffffffff80000000+)

For virtual address 0xffffffff81000000:

Level	Index Calculation	Index	Maps To
PML4	(0xffffffff81000000 >> 39) & 0x1FF	511	PDPT_HIGH at 0x3000
PDPT	(0xffffffff81000000 >> 30) & 0x1FF	510	PD at high_pd_base
PD	(0xffffffff81000000 >> 21) & 0x1FF	8	Physical 8 × 2MB = 0x1000000 ✅

The mapping is correct:

• 0xffffffff80000000 → physical 0x0
• 0xffffffff81000000 → physical 0x1000000 ✅

---

4. RSI Register Setup

In vcpu.rs:

let regs = kvm_regs {
    rip: kernel_entry,      // Entry point (virtual address)
    rsi: boot_params_addr,  // Boot params pointer (Linux boot protocol)
    rflags: 0x2,
    rsp: 0x8000,
    ..Default::default()
};

RSI correctly points to boot_params_addr (0x7000). ✅

---

5. Constants Inconsistency

mod.rs layout module:

pub const PVH_START_INFO_ADDR: u64 = 0x7000;  // Used
pub const ZERO_PAGE_ADDR: u64 = 0x10000;      // NOT USED - misleading!

linux.rs:

pub const BOOT_PARAMS_ADDR: u64 = 0x7000;  // Used

The ZERO_PAGE_ADDR constant is defined but never used, which is confusing since "zero page" is another name for boot_params in Linux terminology.

---

Applied Fixes

Fix 1: Relocated Boot Structures ✅

Moved all boot structures above the page table area (0xA000 max):

Structure	Old Address	New Address	Status
BOOT_PARAMS_ADDR	0x7000	0x20000	✅ Already done
PVH_START_INFO_ADDR	0x7000	0x21000	✅ Fixed
E820_MAP_ADDR	0x9000	0x22000	✅ Fixed
CMDLINE_ADDR	0x8000	0x30000	✅ Already done
BOOT_STACK_POINTER	0x8FF0	0x1FFF0	✅ Fixed

Fix 2: Updated vcpu.rs ✅

Changed hardcoded stack pointer from 0x8000 to 0x1FFF0:

• File: vmm/src/kvm/vcpu.rs
• Stack now safely above page tables but below boot structures

Fix 3: Added Layout Documentation ✅

Updated mod.rs with comprehensive memory map documentation:

0x0000 - 0x04FF  : Reserved (IVT, BDA)
0x0500 - 0x052F  : GDT (3 entries)
0x1000 - 0x1FFF  : PML4
0x2000 - 0x2FFF  : PDPT_LOW (identity mapping)
0x3000 - 0x3FFF  : PDPT_HIGH (kernel high-half mapping)
0x4000 - 0x7FFF  : PD tables for identity mapping (up to 4 for 4GB)
0x8000 - 0x9FFF  : PD tables for high-half kernel mapping
0xA000 - 0x1FFFF : Reserved / available
0x20000          : boot_params (Linux zero page) - 4KB
0x21000          : PVH start_info - 4KB
0x22000          : E820 memory map - 4KB
0x30000          : Boot command line - 4KB
0x31000 - 0xFFFFF: Stack and scratch space
0x100000         : Kernel load address (1MB)

Verification Results ✅

All memory sizes from 128MB to 16GB now pass without overlaps:

Memory: 128 MB  - Page tables: 0x1000-0x6FFF ✅
Memory: 512 MB  - Page tables: 0x1000-0x6FFF ✅
Memory: 1024 MB - Page tables: 0x1000-0x6FFF ✅
Memory: 2048 MB - Page tables: 0x1000-0x7FFF ✅
Memory: 4096 MB - Page tables: 0x1000-0x9FFF ✅
Memory: 8192 MB - Page tables: 0x1000-0x9FFF ✅
Memory: 16384 MB- Page tables: 0x1000-0x9FFF ✅

---

Verification Checklist

• ELF segments loaded to correct physical addresses
• Entry point is virtual address (handled by page tables)
• RSI contains boot_params pointer
• High-half mapping: 0xffffffff80000000 → physical 0
• High-half mapping: 0xffffffff81000000 → physical 0x1000000
• Memory layout has no overlaps ← FIXED
• Constants are consistent and documented ← FIXED

Files Modified

1. vmm/src/boot/mod.rs - Updated layout constants, added documentation
2. vmm/src/kvm/vcpu.rs - Updated stack pointer from 0x8000 to 0x1FFF0
3. docs/MEMORY_LAYOUT_ANALYSIS.md - This analysis document