Apple Silicon and Virtual Machines: Beating the 2 VM Limit

For those wondering what I’ve been doing outside of OpenCore Legacy Patcher the past few months, I had the amazing opportunity to work as a Mac Admin Intern at a local consulting company.

One of the areas I’ve been working on a lot is macOS virtual machines, specifically Apple Silicon virtual machines based on Apple’s virtualization framework. However, after doing a lot of development and testing using Apple’s VM stack (via the amazing project, UTM), I found a very frustrating limitation: an Apple Silicon host can only have a maximum of 2 macOS guest VMs active at once.

This is commonly seen with this error, which is generated by Virtualization.framework: :

The number of virtual machines exceeds the limit. The maximum supported number of active virtual machines has been reached.

The main reason for this error comes from macOS’s SLA, section 2.B.iii:

(iii) within a virtual operating system environment on each Apple-branded computer you own or control that is already running the Apple Software, the Apple Software, or any prior macOS or OS (B) tested during software development; (c) using macOS Server; or (d) personal, non-commercial use.

Although I can’t officially virtualize more than 2 copies of macOS on the same machine for work, I was still interested to find out where Apple embeds these checks in macOS and whether hobbyists and researchers can enable support for more than 2 active macOS VMs simultaneously.

macOS Internals Deep Dive

To start, I initially thought that this limitation was user location based and thus would be inherent somewhere within /System/Library/Frameworks/Virtualization.framework. Due to macOS Big Sur’s Dyld merging of frameworks, we will need to either manually remove the framework or use tooling like Hopper Disassembler to load specific binaries embedded in the Dyld shared cache.

With this, I was able to examine the structure more closely. However, after several hours of research, I have been unable to find where Apple sets the VM limit. At best, I can only determine that the error message was generated from the framework, but nowhere in userspace does Apple define a hardcoded 2 VM limit…

Following a tip from Javinsky on the Hack Different Discord server, I learned that Apple’s guest limits are enforced somewhere within a closed-source part of XNU (the macOS kernel). Although I had no strings to spare, I knew the Intel kernel would not contain similar code. So with such a quick comparison between the functions and strings of the Intel and Apple silicon kernels, I found that the main VM stack is below hv_vm_*.

Upon throwing the development kernel for macOS Sonoma Beta 4 (23A5301h) into IDA, I found the init code for the VM stack: hv_init(): :

Here we see how Apple handles VM limits: Using int hv_apple_isa_vm_quota Variables increment/decrement kernel variables when starting/stopping new virtual machines:

growth function	salary increase ceremony
void hv_vm_destroy_0(hv_vm_t_0 *vm)	void hv_trap_vm_create(uint64_t_0 arg)

And something else interesting, 2 new boot-args: hypervisor= And hv_apple_isa_vm_quota=.

There is a simple gate check for the first one, which is far more interesting: hv_apple_isa_vm_quota= Can override VM limits in the kernel!

However, after some further research, I found that this logic is not the same in the release kernel. Instead, Apple swapped hypervisor boot-args with one AppleInternal Check through System Integrity Protection:

/*
  CSR_ALLOW_APPLE_INTERNAL = 0x10

  From XNU Source:
  #define CSR_ALLOW_APPLE_INTERNAL (1 << 4)
  https://opensource.apple.com/source/xnu/xnu-7195.121.3/bsd/sys/csr.h.auto.html
*/
if ((*(int8_t *)_csr_config & 0x10) != 0x0) {
  _PE_parse_boot_argn_internal(*0xfffffe00072696d0 + 0x6c, "hv_apple_isa_vm_quota", 0xfffffe0007b58410, 0x4, 0x0);
}

Here we have 2 options:

1. Boot Apple’s development kernel
2. Modify release kernel to strip AppleInternal check

To save what little sanity I have left, we’ll go with the first option. So now my next challenge: booting a development kernel on my MacBook Pro.

Building a development kernel repository

To create a development kernel archive, we will need to fetch the appropriate kernel debug kit from Apple’s developer site. Note that KDK Sure Match the host, otherwise, problems may occur with kernel and kext linkage as well as during boot.

Once you’ve downloaded the KDK disk image and installed the embedded packages, next check what type of kernel your Mac uses:

uname -v | awk -F "https://khronokernel.com/" '{print $NF}'| awk -F '_' '{print $NF}'

On M2 Pro MacBook Pro (Mac14,9), it will return T6020. On other CPU models, especially on different generations such as M1 vs M2, the kernel version will be different:

Now we can start building our kernel!

Assumes the following invocation:

• uses host machine T6020 kernels
• The host is running macOS 14.0, build 23A5301h

Make sure you adjust your invocations below to match your host respectively.

• Much appreciation, especially to Apple’s engineers. Jeremy C. Andrus For their blog post on booting a custom kernel:

sudo kmutil create \
  --arch arm64e \
  --no-authorization \
  --variant-suffix development \
  --new boot \
  --boot-path VirtualMachine.kc \
  --kernel /Library/Developer/KDKs/KDK_14.0_23A5301h.kdk/System/Library/Kernels/kernel.development.t6020 \
  --repository /Library/Developer/KDKs/KDK_14.0_23A5301h.kdk/System/Library/Extensions \
  --repository /System/Library/Extensions \
  --repository /System/Library/DriverExtensions \
  --explicit-only $(kmutil inspect -V release --no-header | grep -v "SEPHiber" | awk '{print " -b "$1; }')

this will create a VirtualMachine.kc File in your home directory. Take note of the path, as we will need to access it from RecoveryOS.

configuring our mac to boot the development kernel collection

Finally turn off your Mac, and boot into recovery by pressing the power button and selecting “Options”:

Next, authorize the user, and select Utilities -> Terminal from the menubar. Here we will set some policies for our machine:

1. Disable System Integrity Protection
2. Allow custom boot args to be passed
3. Configure (adjust) our Mac to boot our custom kernel archive. Macintosh HD for your volume)
4. set our boot-args
   • kcsuffix=:set kernel collection variant at boot
   • hypervisor=:Enable special features in the virtualization stack (ie VM quota override)
   • hv_apple_isa_vm_quota=: Override VM quota, maximum value is 0x7FFFFFFF (set to 0xFF (255)VM for practicality)

csrutil disable
bputil --disable-boot-args-restriction
kmutil configure-boot --volume /Volumes/Macintosh\ HD --custom-boot-object /Volumes/Macintosh\ HD/Users/*/VirtualMachine.kc
nvram 40A0DDD2-77F8-4392-B4A3-1E7304206516:boot-args="kcsuffix=development hypervisor=0x1 hv_apple_isa_vm_quota=0xFF"

Once rebooted, you can verify this by executing in the terminal:

sysctl kern.osbuildconfig
nvram boot-args

Putting our machine to work!

Now that everything is ready, you will now want to run any virtualization solution using the Virtualization.Framework. Some examples include:

Now we can activate our VM! Below I’ve got 9 macOS VMs running simultaneously on my M2 Pro MacBook Pro, and still usable for testing!

(This was also the first time I heard the fan turn on on this machine, so we know we’re getting our money’s worth ;p)

When did Apple provide us with this feature?

It seems that with macOS 12, Monterey, Apple has added this boot-args to the virtualization stack. And as we saw with Kernel of Sonoma, AppleInternal Cheque still exists in Monterey, too. It looks like Apple still has a lot of secrets hidden within XNU.

Our work is being undone for OS updates

When using custom kernel collections with Apple silicon, there are some unfortunate drawbacks. On top of that, streamlined OS updates are no longer available. You can still install the update, although your machine will error out upon completing the update.

To fix this, you will need to revert to the stock kernel archive on your machine.

To reset, you just need to create a new boot policy bputil In RecoveryOS. This can happen either with full security (--full-security) or any other combination (e.g. --disable-boot-args-restriction). Once you run it in RecoveryOS and reboot, the stock kernel should take effect.

closing thoughts

Overall this was a really interesting research journey and I’m glad I was able to figure out how Apple implemented this limitation. Additionally, I really appreciate that even though this is an unsupported use case, the virtualization team at CoreOS still provided enthusiasts with the option to override this limitation (even if not documented or easy to do so).

I have some improvements in mind for the future (although these are unlikely to be implemented):

• Develop tooling to automate KC building and booting.
  • Download and generate a development kernel archive for a given host.
  • Configure the host in RecoveryOS to boot the kernel archive.
• Consider developing a kernel extension that could override this hv_apple_isa_vm_quota Variable.
  • Removes the need for a custom,development kernel archive.

Otherwise I hope the community finds this blog post interesting, my next visit will probably be to see if DEP enrollment/serial number override is possible for Apple Silicon VMs. Might not be as lucky as this post though ;P