Detecting and Mitigating Hidden Virtual Machines (T1564.006)

Security tools watch the host. A virtual machine running on that host is a separate computer they cannot see into. Run Virtual Instance (T1564.006) is a Stealth technique, listed by MITRE for ESXi, Linux, macOS and Windows. It is a subtechnique of Hide Artifacts.

How attackers use it

MITRE explains that by running malicious code inside a virtual instance, adversaries can hide artifacts of their behavior from security tools that cannot monitor inside it. Depending on the virtual network setup, such as a bridged adapter, traffic from the virtual instance can be hard to trace back to the compromised host, because the IP address and hostname might not match known values.

They may use native virtualization such as Hyper-V, lightweight emulators such as QEMU, or drop the files needed to run a virtual instance, as with VirtualBox binaries. They may create a shared folder between guest and host with permissions that let the instance touch the host file system.

MITRE also describes temporary virtual environments such as Windows Sandbox, which uses .wsb configuration files to define execution parameters. In those files, the MappedFolder property creates a shared folder and the LogonCommand property names a payload.

On VMware, MITRE says adversaries may use the vCenter console to create new VMs, or create them directly on ESXi servers by running a valid .vmx file with the /bin/vmx utility. Adding that command to /etc/rc.local.d/local.sh makes the VM restart persistently. A VM created this way does not appear in the vCenter console or in the output of vim-cmd vmsvc/getallvms on the ESXi server, hiding it from typical administrative activities.

How to detect it

MITRE's analytics, by platform:

• Windows. Unusual execution of virtualization binaries such as VBoxManage.exe, vmware-vmx.exe or vmwp.exe with headless or suppressed notification arguments, and registry and service changes linked to virtualization installs.
• Linux. QEMU, KVM or VirtualBox processes with unusual flags such as nographic or snapshot, and VM images created in atypical directories.
• macOS. Parallels, VMware Fusion or VirtualBox run with arguments that hide the UI, and plist modifications that indicate hidden virtualization.
• ESXi. Direct execution of /bin/vmx, rogue .vmx files not registered in the vCenter inventory, and edits to /etc/rc.local.d/local.sh for persistence.

The question to ask of every virtualization process is simple: did we deploy this?

How to reduce the risk

MITRE lists three mitigations.

1. Disable or remove feature or program. Turn off native virtualization such as Hyper-V where it is not needed. Consider disabling Windows Sandbox if it is not needed to test or debug applications.
2. Audit. Periodically audit virtual machines for abnormalities. On ESXi servers, periodically compare the output of vim-cmd vmsvc/getallvms, which lists VMs, against the VMs actually running on the host. MITRE names an esxcli vm process list comparison for this.
3. Execution prevention. Use application control to block installation and use of unapproved virtualization software.

Why inventory is the real control

Most of what MITRE lists for this technique is about knowing what should be there. A virtual machine that nobody deployed is, by definition, a difference between your records and reality. On ESXi that difference is detectable with two commands, as shown above. On endpoints it shows up as virtualization software or features that your build never included.

Network behavior supports the same conclusion. MITRE notes that traffic from a VM on a bridged adapter may carry an address or hostname that does not match known values. So a device on your network with an unfamiliar name and a hardware address that belongs to a virtualization vendor's range is a prompt to ask which physical machine hosts it. Your DHCP and switch tables are good places to look.

If you find an unapproved VM

1. Note the host, the process, the image file location and any shared folders configured between guest and host.
2. Capture what you can before shutting anything down, including the configuration file and the network settings.
3. Check how persistence was set up, such as startup scripts on ESXi or scheduled tasks on endpoints.
4. Review the host for the access that allowed it, since creating a VM needs privileges.
5. Remove the VM and the virtualization software if it is not approved, and add application control so it cannot return.

Check your own environment

Run these read-only checks where they apply.

# Windows (PowerShell): is Hyper-V or Windows Sandbox enabled on this machine?
Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All
Get-WindowsOptionalFeature -Online -FeatureName Containers-DisposableClientVM

# ESXi shell: VMs registered on this host
vim-cmd vmsvc/getallvms

# ESXi shell: VMs actually running on this host
esxcli vm process list

Compare the last two lists. A running VM that is not in the registered list is exactly the case MITRE describes.

Then ask:

• Which endpoints have virtualization features enabled, and do they need them?
• Does application control block VirtualBox, QEMU and similar tools from user-writable folders?
• Does your asset inventory include virtual machines, and do you reconcile it against hypervisor reality?

Finding machines nobody listed is an inventory problem as much as a detection problem. The demo below shows how asset discovery surfaces systems that are missing from the list.

Click through it at your own pace, or open it full screen.

Common mistakes

• Assuming the hypervisor console is the whole truth. On ESXi, MITRE describes a way for a VM to run without appearing in it.
• Leaving Hyper-V and Windows Sandbox enabled on every endpoint because the image was built that way.
• Auditing virtual machines once a year. The comparison is cheap, so make it routine.

Quick check

Sources

• MITRE ATT&CK T1564.006: Run Virtual Instance