Nested Virtualization on Windows 11: The VBS Conflict Explained

# microsoft# security# tooling# tutorial
Nested Virtualization on Windows 11: The VBS Conflict ExplainedStefor07

Introduction After upgrading to a modern system with full virtualization support —...

Introduction

After upgrading to a modern system with full virtualization support — including VT-x, EPT, IOMMU, TPM 2.0, and Secure Boot — I expected everything to work flawlessly. Instead, I encountered crashes and instability when running nested virtualization scenarios, particularly when attempting to run VMware inside VMware Workstation Pro.

At first, I suspected hardware instability, BIOS misconfiguration, or a VMware bug. However, after extensive testing, I discovered the real culprit: Windows Virtualization-Based Security (VBS).

This article covers:

  • How VBS affects desktop hypervisors on modern Windows 11 (24H2/25H2)
  • Why nested virtualization fails under certain configurations
  • Practical solutions for advanced lab environments requiring full VT-x access

The Problem: VMware Nested Virtualization Failures

The issue surfaced when I upgraded to a 14th generation Intel CPU with hybrid cores (P-cores and E-cores). While VMware Workstation Pro ran perfectly on the host, problems arose when I tried running VMware inside a Windows 11 24H2 VM.

Attempting to boot a Windows 11 ISO inside the nested VM resulted in an immediate crash: VMware reported that an exception had caused a breakpoint, and the VM shut down.

Understanding the Root Cause: VBS

Virtualization-Based Security (VBS) is fully integrated into Windows 11's kernel and takes exclusive control of VT-x/AMD-V. This prevents desktop hypervisors from accessing hardware virtualization directly, causing conflicts with nested virtualization.

Initial Troubleshooting Attempts

I tried several manual approaches to disable VBS:

  1. Disabling Core Isolation (Windows Security → Device Security)
  2. Disabling hypervisor launch: 
   bcdedit /set hypervisorlaunchtype off
Enter fullscreen mode Exit fullscreen mode
  1. Group Policy and Registry edits

None of these methods worked completely.

The Working Solution: Microsoft's DG_Readiness_Tool

I discovered Microsoft's official script for disabling VBS:

Download: DG_Readiness_Tool

Usage:

DG_Readiness_Tool_v3.6.ps1 -Disable
Enter fullscreen mode Exit fullscreen mode

After running the script and rebooting, VBS is successfully disabled.

Important: This change is temporary — Windows re-enables VBS after subsequent reboots, so the script must be rerun when needed.

The Role of HypervisorIOMMUPolicy

During testing, I discovered that HypervisorIOMMUPolicy plays a crucial role in nested virtualization stability.

Disabling HypervisorIOMMUPolicy

To disable it, run in elevated PowerShell:

bcdedit /set HypervisorIOMMUPolicy off
Enter fullscreen mode Exit fullscreen mode

Restart your computer for the change to take effect.

What This Achieves

Disabling HypervisorIOMMUPolicy prevents the guest VM from receiving full IOMMU exposure, which:

  • ✅ Allows lightweight hypervisors and Android emulators to run reliably inside VMs
  • ❌ Does NOT solve VMware-inside-VMware scenarios — full VBS disabling is still required

Why This Happens

  1. Virtualized IOMMU conflicts: Not all hypervisors handle virtualized IOMMU correctly, leading to crashes
  2. VMware's requirements: Running VMware as a guest requires full EPT/NPT and IOMMU support
  3. Security feature interaction: VBS + IOMMU virtualization creates a complex layering that VMware nested virtualization cannot navigate without full VBS disabling

Complete Solutions

Since VBS cannot be permanently removed from Windows 11 24H2/25H2, here are your options:

Solution 1: Temporary VBS Disable

  • Run DG_Readiness_Tool before each session requiring nested VMware
  • Provides full VT-x access temporarily
  • Limitation: Must be rerun after reboots

Solution 2: Downgrade or Dual-Boot

  • Use Windows 11 22H2 (doesn't enforce VBS)
  • Install as dual-boot alongside 24H2/25H2
  • Provides permanent, stable hardware virtualization

Solution 3: Switch to Linux

  • No VBS restrictions
  • Superior virtualization performance
  • Recommended for advanced lab environments
  • Popular options: Ubuntu, Fedora, Debian, Arch-based distributions

Compatibility Matrix

The table below shows which hypervisors work under different configurations when running inside VMware VMs:

Hypervisor / Software HypervisorIOMMUPolicy ON HypervisorIOMMUPolicy OFF VBS Disabled Notes
VMware Workstation Pro ❌ Crash ❌ Unstable ✅ Works Requires full VBS disable
VirtualBox ✅ Works ✅ Works ✅ Works Reliable in all configurations
Android Emulators (AVD) ❌ May fail ✅ Works ✅ Works Best with HypervisorIOMMUPolicy OFF
KVM (Linux VM) ❌ Cannot run ✅ Works ✅ Works Requires HypervisorIOMMUPolicy OFF
Hyper-V (Windows VM) ❌ Not supported ❌ Limited ❌ Limited Nested Hyper-V not functional
Lightweight hypervisors ❌ May fail ✅ Works ✅ Works Depends on VT-x exposure

Legend: ✅ Works reliably | ❌ Fails or unstable

Key Takeaways

  1. For lightweight hypervisors/emulators: Disable HypervisorIOMMUPolicy only 
   bcdedit /set HypervisorIOMMUPolicy off
Enter fullscreen mode Exit fullscreen mode
  1. For full VMware nested virtualization: Disable both HypervisorIOMMUPolicy AND VBS 
   DG_Readiness_Tool_v3.6.ps1 -Disable
Enter fullscreen mode Exit fullscreen mode

  1. Hardware requirements:

    • Enable VT-x/AMD-V in BIOS/UEFI
    • Expose virtualization to guest VM in VMware settings
    • Use multi-core CPU allocation

  2. Long-term solutions:

    • Consider Windows 11 22H2 for permanent VBS-free environment
    • Linux hosts provide best performance and stability for complex nested setups

Conclusion

Modern Windows 11's VBS and IOMMU policies create significant challenges for nested virtualization. While these security features protect system integrity, they limit hardware virtualization access.

By understanding the interaction between VBS, HypervisorIOMMUPolicy, and VT-x exposure, advanced users can:

  • Run lightweight hypervisors and emulators reliably with minimal changes
  • Achieve full nested VMware functionality through temporary VBS disabling
  • Make informed decisions about host OS choice for lab environments

For users requiring frequent nested virtualization, Linux hosts or Windows 11 22H2 in dual-boot offer the most practical, stable solutions without ongoing workarounds.