Troubleshooting

Comprehensive troubleshooting guide for QuantaVirt — diagnostic tools, boot failures, VM issues, CPU virtualization, memory, storage, networking, PQC/QUAC 100 problems, performance degradation, and complete error code reference.

Troubleshooting Overview #

When diagnosing QuantaVirt issues, follow this general approach: check system logs first, run the built-in self-test, verify hardware requirements are met, then narrow down to the specific subsystem. Most issues fall into one of the categories below.

Diagnostic Tools #

# Full system self-test
quantavirt system self-test
# [PASS]  CPU: Intel VMX supported and enabled
# [PASS]  IOMMU: Intel VT-d active (DMAR table found)
# [PASS]  Memory: 64 GB available, hugepages configured (32 × 2MB)
# [PASS]  Storage: /var/lib/quantavirt writable, 500 GB free
# [PASS]  PQC: ML-KEM-768 keygen/encaps/decaps OK (1.2 ms software)
# [PASS]  QUAC 100: PCIe device 0x1DB7:0x0100 online, firmware 1.0.0
# [PASS]  API: REST listening on 127.0.0.1:8400
# [PASS]  gRPC: listening on 127.0.0.1:8401
# Result: 8/8 passed, 0 warnings, 0 failures

# System information dump
quantavirt system show

# VM-specific diagnostics
quantavirt vm diagnostics <vm-name>

# Real-time log streaming
quantavirt log tail --level debug
quantavirt log tail --subsystem pqc
quantavirt log tail --vm <vm-name>

# Hardware capability check
quantavirt system capabilities
# CPU:       Intel Core i9-13900K (VMX, EPT, VPID, unrestricted guest)
# IOMMU:     Intel VT-d (interrupt remapping, ATS)
# SEV:       Not available (AMD only)
# TDX:       Not available (requires Xeon 4th Gen+)
# QUAC 100:  1 device, firmware 1.0.0, BAR0 0xfb000000 (64 KB)

# Performance counters
quantavirt stats <vm-name> --interval 1s

Boot Failures #

Hypervisor fails to boot — "VMX not supported"

Symptom: Hypervisor kernel panics at boot with FATAL: VMX/SVM not available.

Causes & Fixes:

Hypervisor fails to boot — "IOMMU not found"

Symptom: Warning IOMMU: No DMAR/IVRS table found at boot. PCI passthrough and QUAC 100 DMA will not work.

Fixes:

GRUB / systemd-boot fails to load hypervisor

Symptom: Bootloader menu appears but selecting QuantaVirt produces error: invalid magic number or blank screen.

VM Start / Stop Issues #

"Failed to create VMCS" / "VMCB allocation failed"

Cause: Insufficient contiguous physical memory for the 4 KB VMCS/VMCB structure, or VMX/SVM context limit reached.

# Check available memory
quantavirt system show | grep -i memory

# Check running VM count vs limits
quantavirt vm list --all | wc -l

# Free memory by stopping unused VMs
quantavirt vm stop <unused-vm>

"EPT/NPT setup failed"

Cause: Unable to allocate 4-level page tables for VM memory mapping. Typically caused by memory fragmentation or insufficient hugepages.

# Check hugepage availability
cat /proc/meminfo | grep -i huge

# Allocate more hugepages (2 MB pages)
echo 4096 > /proc/sys/vm/nr_hugepages

# Or use kernel parameter for boot-time allocation
# quantavirt.hugepages=4096

VM stuck in STOPPING state

Cause: Guest not responding to ACPI power button event. Guest agent may be hung or ACPI not installed in guest.

# Force stop (equivalent to power cut)
quantavirt vm stop <vm-name> --force

# If still stuck, destroy and recreate
quantavirt vm destroy <vm-name>

# Prevent in future: install ACPI support in guest
# Ubuntu/Debian: apt install acpid
# RHEL/Fedora:   dnf install acpid

Guest boots to blank screen / no display

CPU Virtualization Issues #

Guest crashes with "Invalid opcode" (#UD)

Cause: Guest using CPU instructions not exposed through CPUID masking. Common when migrating VMs between hosts with different CPU generations.

# Check CPU features exposed to guest
quantavirt vm show <vm-name> --cpu-features

# Use compatible CPU model instead of host passthrough
"cpu": { "model": "Skylake-Server", "count": 4 }

# Or explicitly disable problematic features
"cpu": { "model": "host", "features": ["-avx512f", "-avx512bw"] }

Poor vCPU performance / high steal time

Nested virtualization not working

# Verify nested virt is enabled (must be set at hypervisor boot)
quantavirt system show | grep nested
# nested_virtualization: enabled (beta)

# If disabled, add kernel parameter:
# quantavirt.nested=on

# Expose VMX/SVM to guest
"cpu": { "model": "host", "features": ["+vmx"] }

Memory Issues #

"Insufficient memory" when starting VM

# Check host memory
quantavirt system show | grep -A5 memory

# Check per-VM memory allocation
quantavirt vm list --format wide
# NAME            STATE    vCPUs  RAM     DISK
# database        running  16     64G     500G
# webserver       running  4      8G      50G
# new-vm          created  2      32G     ← won't fit if only 20G free

# Enable memory balloon to reclaim unused guest memory
"memory": { "size": "8G", "balloon": true }

# Reduce balloon in running VM (give memory back to host)
quantavirt vm set <vm-name> --memory-balloon 4G

Hugepage allocation failures

# Check hugepage stats
cat /proc/meminfo | grep Huge
# HugePages_Total:    2048
# HugePages_Free:     128
# HugePages_Rsvd:     0
# Hugepagesize:       2048 kB

# If Free is too low, allocate more (must be done early after boot)
echo 4096 > /proc/sys/vm/nr_hugepages

# For 1 GB pages (best for large VMs), use kernel param:
# quantavirt.hugepages=64 quantavirt.hugepagesz=1G

# Fall back to non-hugepage if needed
"memory": { "size": "16G", "hugepages": false }

Storage Issues #

"Disk image locked" / concurrent access error

Cause: Another VM or process holds an exclusive lock on the disk image. QuantaVirt prevents concurrent write access to the same image to avoid corruption.

# Check which VM holds the lock
quantavirt storage show <disk-name>
# locked_by: database (pid 4521)

# Force-release a stale lock (use with extreme caution!)
quantavirt storage unlock <disk-name> --force

QCOW2 corruption detected

# Check QCOW2 integrity
quantavirt storage check <disk-name>
# Checking refcounts... 3 errors found
# Checking L1/L2 tables... OK

# Repair (fixes refcount and metadata errors)
quantavirt storage repair <disk-name>

# Repair with leak fix (reclaims unreferenced clusters)
quantavirt storage repair <disk-name> --fix-leaks

Slow disk I/O

Network Issues #

VM has no network connectivity

# Step 1: Verify network exists and is active
quantavirt network show default

# Step 2: Check VM NIC configuration
quantavirt vm show <vm-name> | grep -A10 network

# Step 3: Check DHCP lease was assigned
quantavirt network dhcp-list default

# Step 4: Verify bridge interface is up on host
ip link show qvbr0

# Step 5: Check firewall rules aren't blocking
quantavirt network rule list default

Low network throughput

PQC & QUAC 100 Issues #

QUAC 100 not detected

# Check PCIe bus
lspci | grep 1DB7
# Should show: XX:XX.0 Encryption controller: Dyber Inc. QUAC 100 (rev 01)

# If not found:
# 1. Verify card is seated in PCIe x8/x16 slot
# 2. Check PCIe power connector (if applicable)
# 3. Try different PCIe slot
# 4. Update motherboard BIOS

# Check driver loaded
quantavirt system show | grep -i quac
# quac100: online, firmware 1.0.0, BAR0 0xfb000000

# If driver not loaded:
quantavirt quac100 probe

QUAC 100 firmware update failures

# Check current firmware version
quantavirt quac100 info
# Vendor:    Dyber, Inc. (0x1DB7)
# Device:    QUAC 100 (0x0100)
# Firmware:  1.0.0
# Serial:    QV-100-XXXXXX

# Firmware update (requires no running VMs using PQC)
quantavirt quac100 firmware-update /path/to/quac100-fw-1.1.0.bin

# If update fails mid-flight, the QUAC 100 has dual-bank firmware:
# it will revert to the previous working firmware on next reboot.
# Power cycle the host if the device becomes unresponsive.

PQC crypto self-test failures

Software fallback engaged unexpectedly

# Check which backend is active
quantavirt pqc status
# Backend:       software (QUAC 100 not available)
# ML-KEM-768:    2.1 ms/op (software)
# ML-DSA-65:     4.3 ms/op (software)

# Diagnose why QUAC 100 is unavailable
quantavirt quac100 info
# If "device not found" → see QUAC 100 not detected above
# If "firmware error" → update firmware
# If "busy" → another process has exclusive access

# Force hardware backend (will error if device unavailable)
"pqc": { "backend": "quac100" }

Performance Problems #

General performance diagnostics

# Real-time VM performance metrics
quantavirt stats <vm-name> --interval 1s
# vCPU:    45% (user 38%, kernel 7%)
# VM-Exits: 12,400/sec (I/O: 8,200, CPUID: 2,100, EPT: 1,800, other: 300)
# Memory:  4.2 GB / 8.0 GB (balloon: 0 MB)
# Disk:    read 450 MB/s, write 320 MB/s (IOPS: 85K r / 42K w)
# Net:     rx 2.1 Gbps, tx 1.8 Gbps (packets: 180K/s)

# Host-level overview
quantavirt stats --host --interval 1s

Excessive VM-exits

GUI Issues #

GUI cannot connect to hypervisor

# Check API is listening
quantavirt system show | grep -i api
# api.rest:  127.0.0.1:8400 (active)
# api.grpc:  127.0.0.1:8401 (active)

# If remote GUI, ensure API binds to accessible address:
# /etc/quantavirt/quantavirt.conf
# api.listen = 0.0.0.0:8400

# Test connectivity from GUI machine
curl -s http://<hypervisor-ip>:8400/api/v1/system | jq .

# Firewall may block — allow port 8400/8401 on host

VNC console blank or frozen

API & CLI Issues #

CLI returns "Connection refused"

# CLI communicates with hypervisor via Unix socket or REST API

# Check if hypervisor is running
systemctl status quantavirt-hypervisor

# Check socket exists
ls -la /var/run/quantavirt/quantavirt.sock

# Try explicit API endpoint
quantavirt --api http://127.0.0.1:8400 system show

# If managing remote host
export QUANTAVIRT_API=http://192.168.1.100:8400
quantavirt system show

API authentication errors

# Generate new API token
quantavirt auth token-create --name admin-token --role admin
# Token: qvt_a1b2c3d4e5f6...

# Use token with CLI
export QUANTAVIRT_TOKEN=qvt_a1b2c3d4e5f6...
quantavirt vm list

# Use token with REST API
curl -H "Authorization: Bearer qvt_a1b2c3d4e5f6..." \
  http://127.0.0.1:8400/api/v1/vms

Migration Failures #

"PQC key exchange failed" during migration

Cause: Source and destination hosts cannot establish a PQC-encrypted channel for VM transfer. Both hosts must have compatible PQC backends.

# Verify PQC on both hosts
quantavirt pqc status                          # source
ssh dest-host quantavirt pqc status            # destination

# Both must support the same KEM algorithm
# If one has QUAC 100 and other doesn't, both will use software fallback

# Force software backend for compatibility
quantavirt vm migrate <vm> --dest <host> --pqc-backend software

"CPU incompatible" migration error

# Check CPU feature delta
quantavirt migrate check <vm> --dest <host>
# [FAIL] Destination missing features: avx512f, avx512bw
# [PASS] Memory: 64 GB available on destination
# [PASS] Storage: shared NFS mount verified

# Fix: use a compatible CPU model instead of host passthrough
quantavirt vm set <vm> --cpu-model Skylake-Server
# Then retry migration

Migration stalls / never completes

Log Reference #

# Set log level (trace, debug, info, warn, error)
quantavirt system set logging.level debug

# Tail all logs
quantavirt log tail --level debug

# Filter by subsystem
quantavirt log tail --subsystem vmx
quantavirt log tail --subsystem ept
quantavirt log tail --subsystem pqc
quantavirt log tail --subsystem virtio-blk

# Filter by VM
quantavirt log tail --vm ubuntu-server

# Export logs for support
quantavirt system support-bundle --output /tmp/qv-support.tar.gz

Error Code Reference #

Hypervisor Core Errors (QV_ERR_*)

PQC Subsystem Errors (PQC_ERR_*)

QUAC 100 Hardware Errors (QUAC100_ERR_*)

Getting Support #

If you cannot resolve an issue using this guide, collect a support bundle and contact Dyber support.

# Generate support bundle (includes logs, config, hardware info)
quantavirt system support-bundle --output /tmp/qv-support.tar.gz

# Bundle contents:
#   system-info.json        — hardware, firmware, OS details
#   hypervisor.log          — last 10,000 lines
#   pqc.log                 — last 10,000 lines
#   vm/*.log                — per-VM logs
#   quantavirt.conf          — hypervisor configuration (secrets redacted)
#   vm-configs/*.json        — VM configurations (passwords redacted)
#   self-test-results.json   — latest self-test output
#   dmesg.log               — kernel ring buffer
#   lspci.txt               — PCIe device enumeration