Your Hypervisor Is a Neon Sign

2026-08-02

I keep seeing the same line in hypervisor forks:

self.exit_entry_hw_overhead = 420;

Someone measured this on a laptop in 2019. It got copied into a thousand repos since. Nobody re-measured it. That is the whole problem in one constant.

Real CPUID on bare metal runs 100 to 250 cycles. No trap, no handler, just silicon doing its job. Your hypervisor traps it, drops to root mode, runs a handler, maybe calls real CPUID, masks some bits, resumes the guest. That whole path costs 500 to 900 cycles. Bare metal says 180. You hardcode 600. Sample this a few thousand times and plot it. Bare metal gives you one clean mountain. Your hypervisor gives you two. That gap is visible on a bar chart to anyone who bothers to look.

The number is wrong by design

420 is not close to the truth on any given moment. Overhead is not a constant, it is a cloud:

turbo hot        ~380
steady           ~520
P-state          ~640
C-state wake     720+
your constant    420  (wrong every time, both directions)

Subtracting a fixed number is worse than subtracting nothing. Natural timing noise looks random. A fixed subtraction turns that noise into a signature, because nothing else on the machine produces exactly that offset.

The fix is a clamped exponential moving average, tracked per physical core:

if burnt > 100 && burnt < 2000 {
    ovh[pcpu] = (ovh[pcpu] * 7 + burnt) >> 3;
}

Drop the samples outside that range instead of forcing them into the average. A C-state wake spike at 5000 cycles smeared into your EMA still drags your baseline away from the real steady state. And one number for all cores is still wrong. SMT siblings are not interchangeable, contention from the sibling thread shifts timing by 30 to 80 cycles depending on what it’s doing.

Two clocks, one liar

Say you fix TSC offsetting so RDTSC lies convincingly. RDPMC still tells the truth, because performance counters do not read your VMCS TSC offset field. They tick against real hardware.

RDTSC  delta 200   (you lied)
RDPMC  delta 600   (silicon did not)

Run both in a loop and compare the ratio. If it doesn’t match native, you’re caught. The fix is one timeline, not two independent lies:

let gtsc = host_tsc.wrapping_add(vcpu.tsc_offset());

Derive your PMC shadow increments from this same virtual TSC. Every clock the guest can read needs to agree with every other clock, or the mismatch is the tell.

RDPMC gives you two free mistakes

Bit 30 in ECX selects fixed-function performance counters. If your handler only checks ecx < 8 and ignores that bit, you either return garbage or fault on 0x40000000 where real hardware just works. That’s mistake one, and it costs a single instruction to catch.

Mistake two is subtler. Fixed counters freeze when IA32_FIXED_CTR_CTRL or IA32_PERF_GLOBAL_CTRL disable them. Read a frozen counter twice, get the same value both times. If your shadow counter always advances regardless of the enable bits, two RDPMC calls back to back expose you:

rdpmc          ; ecx=0x40000000
mov r8, rax
rdpmc          ; ecx=0x40000000
cmp rax, r8
jne detected

Gate your shadow increment on the actual state the guest wrote to those MSRs. Trapping RDPMC and returning a number that just keeps climbing is not virtualization, it’s a confession.

RDTSCP and the topology check

RDTSCP returns TSC in EDX:EAX and TSC_AUX in ECX. Anti-cheat and legitimate software both use ECX as a cheap core ID, because on real hardware TSC_AUX is set per logical processor to match APIC ID and stays consistent as long as the thread stays pinned.

Pin a thread, read CPUID leaf 0x1F or 0x0B for the expected topology ID, then compare it to what RDTSCP hands back. If your virtualized TSC_AUX is identical across every vCPU, or unrelated to the topology you’re presenting through CPUID, that check fails immediately. Fix it by pulling TSC_AUX from the exact same topology table that answers your CPUID leaves. One source of truth for what a core claims to be.

CPUID’s second hump

Every trapped CPUID that calls real CPUID inside the handler adds 100-plus cycles of your own overhead on top of the real cost. Plot enough samples and you get a second mountain in the histogram that bare metal never produces, because bare metal has exactly one execution path for CPUID, not two.

Precompute the whole table at boot:

fn init_cpuid_table() -> CpuidTable {
    let mut entries = HashMap::new();
    for leaf in KNOWN_LEAVES {
        for subleaf in subleaves_for(leaf) {
            let result = unsafe { real_cpuid(leaf, subleaf) };
            entries.insert((leaf, subleaf), apply_masks(leaf, result));
        }
    }
    CpuidTable { entries }
}

The hot path becomes a hash lookup, nothing more. No real CPUID call at exit time, no second mountain. And if you added an lfence to serialize around a real CPUID call that no longer exists in your hot path, remove it. Leftover instrumentation from an old design isn’t caution, it’s just extra latency nobody needs anymore.

The list

Per-pCPU clamped EMA overhead, kill the fixed constant
One virtual timeline for TSC and PMC, kill the cross-clock ratio check
RDPMC handling with real fixed-counter enable and freeze semantics
TSC_AUX derived from the same topology table as CPUID
Precomputed CPUID table, no nested real hardware calls on the hot path

Fix all five and you get one histogram mountain instead of two, a mean close to bare metal, and clocks that agree with each other under cross-examination.

What comes after this

Once single-primitive histograms are clean, the next layer is joint statistics: RDTSC, RDTSCP, RDPMC, and CPUID measured together against real silicon priors, kurtosis, covariance between primitives, C-state transition correlation. That requires a real measurement corpus per microarchitecture and a long collection window, not a smarter constant. A tighter mean will not save you once someone runs that analysis.

Until then, stop shipping 420.