A virtual machine that ran for months gets shut down for maintenance. You power it back on and it does not come back. The console flashes the VMware logo, the firmware tries one device after another, and you land in the boot manager, at a network boot prompt, or in a loop of restarts. Nothing changed inside Windows. The disks are fine. What changed is the small file next to them that most administrators never look at: the virtual machine's .nvram.
This is the process we use to confirm that the NVRAM is the problem, repair it without touching the guest's disks, and get the machine back with Secure Boot intact.
TL;DR: The.nvramfile is the VM's firmware memory: boot order, UEFI boot entries and the Secure Boot key databases. If it is damaged, missing, or the VM's configuration points at the wrong file, the VM loses track of how to boot. Shut the VM down, keep a copy of the folder, move the old.nvramaside and power on. ESXi writes a clean default file, the firmware finds Windows through the standard fallback path, and you rebuild the boot entry from inside Windows. Check for a virtual TPM before you start, because its state lives in the same file.
What the .nvram file actually holds
Every VM folder on a datastore has a <vmname>.nvram file. It is the virtual equivalent of the settings chip on a physical motherboard. For a VM with UEFI firmware (the default for current Windows and Linux guests) it stores the UEFI variables: the list of boot entries, the order they are tried in, and the Secure Boot databases (PK, KEK, db and dbx). For an older BIOS VM it stores the BIOS settings and boot order.
ESXi creates the file on first power-on and rewrites it whenever the firmware changes a variable. On a UEFI VM it is a fixed size; on the ESXi 8 hosts we manage it is 270,840 bytes. That gives you a quick sanity check before you even open a log.
Two more facts matter for the repair:
- The file the VM uses is named by the
nvramline in its.vmxconfiguration. If that line points at a file that does not exist, ESXi quietly creates a new, empty-default one with that name. It does not warn you. - If the VM has a virtual TPM, the TPM's state is stored in the
.nvramas well. Discarding the file discards the TPM, and anything sealed to it (BitLocker, Windows Hello keys, some credential stores) goes with it.
The symptoms
A damaged or replaced NVRAM shows up at the firmware level, before the operating system has a chance to log anything:
- The VM powers on but drops into the EFI Boot Manager, or tries EFI Network (PXE) and then reports that no compatible boot loader or operating system was found.
- The boot order you set is gone, or a boot entry you created (for example after a P2V conversion or a disk swap) has disappeared.
- The guest starts, VMware Tools reports in for a few seconds, then the VM resets and tries again. You will see it in the logs as a run of short-lived power-ons.
- A Secure Boot failure on a machine whose boot loader has not changed.
- Less often, the power-on itself fails with an error that names the NVRAM.
The tell is that all of this happens with healthy virtual disks. If you mount the disk on another VM, the Windows installation is intact.
Step 1: Look at the VM folder
From the ESXi shell (or the datastore browser in the host client), list the VM's folder:
cd /vmfs/volumes/<datastore>/<vmname>/
ls -la
Check three things:
- Is there a
<vmname>.nvramat all, and is it the normal size? A zero-byte file, a truncated one, or one far off the size of its neighbours is suspect. - Is there an
.nvram-sized file with an odd name? That is the second failure, covered below. - How many
vmware-N.logfiles were written recently? ESXi starts a new log at every power-on. Ten logs in an hour means ten boot attempts, and tells you when the trouble started.
Step 2: Check which NVRAM file the VM is really using
grep -iE 'nvram|firmware|secureboot|vtpm' <vmname>.vmx
A healthy UEFI VM looks like this:
firmware = "efi"
uefi.secureBoot.enabled = "TRUE"
nvram = "<vmname>.nvram"
Here is the one we found on a production server after a maintenance window:
firmware = "efi"
uefi.secureBoot.enabled = "TRUE"
nvram = "TRUE"
The nvram key had been given a boolean instead of a file name, most likely while the Secure Boot setting next to it was being edited. ESXi did exactly what the configuration said: it created a brand-new default NVRAM file called TRUE in the VM folder and booted from that, discarding the machine's real boot entries in the process. The folder listing gives it away: a 270,840-byte file named TRUE and no <vmname>.nvram.
Gotcha: a VM in this state may well boot, because UEFI falls back to the default boot loader path. It only looks fine. The machine is now running on a throwaway firmware store with a meaningless name, and the next person to tidy the folder may delete a file called TRUE as junk. Fix the configuration even if the VM came up.
Step 3: Read the VM's log for the firmware's side of the story
The current power-on is in vmware.log in the same folder. These are the lines worth reading:
grep -E 'DICT +(nvram|firmware)|secureBoot.enabled|SECUREBOOT|Tools: Running status' vmware.log
vmx - DICT firmware = "efi"
vmx - DICT uefi.secureBoot.enabled = "TRUE"
vmx - DICT nvram = "TRUE"
vcpu-0 - SECUREBOOT: Image APPROVED.
vcpu-0 - Tools: Running status rpc handler: 0 => 1.
The DICT lines are the configuration exactly as this power-on read it, so they settle which NVRAM file was used. SECUREBOOT: Image APPROVED means the firmware verified the boot loader. A Tools status that goes to 1 and then times out again a few seconds later, power-on after power-on, is the restart loop.
Do not chase this one: pvnvram: Failed to retrieve timestamp database data. appears on perfectly healthy UEFI VMs with Secure Boot enabled, at every boot. It is not a sign of corruption.
Step 4: Repair
Before you change anything, check for a virtual TPM. If vtpm.present = "TRUE" is in the .vmx, replacing the NVRAM replaces the TPM. Have the BitLocker recovery key for that machine in hand first, or you will not get back into the volume.
- Shut the VM down and confirm it is powered off. The file is held open while it runs.
- Keep a way back. Copy the current NVRAM and configuration:
cp <vmname>.nvram <vmname>.nvram.bak cp <vmname>.vmx <vmname>.vmx.bak - Move the damaged file aside so ESXi creates a clean one:
If the configuration points at the wrong name, as in themv <vmname>.nvram <vmname>.nvram.oldnvram = "TRUE"case, correct it instead of editing the.vmxby hand: in the host client choose Edit settings > VM Options > Advanced > Edit Configuration, setnvramto<vmname>.nvram, and rename the stray file to match if you want to keep it. - Power on. ESXi writes a default NVRAM. On a UEFI VM the firmware then looks for the standard fallback loader on the EFI system partition (
\EFI\BOOT\BOOTX64.EFI), and Windows normally starts. If it stops at the boot manager instead, pick EFI VMware Virtual SCSI Hard Drive (or the NVMe equivalent) by hand. - Rebuild the boot entry from inside Windows, so the next boot does not depend on the fallback path. From an elevated prompt:
bcdedit /enum firmware bcdboot C:\Windows /f UEFIbcdbootrecreates the Windows Boot Manager entry in the new NVRAM and puts it first.
A fresh NVRAM ships with the default Secure Boot keys, so a standard Windows install keeps passing Secure Boot. We confirmed this on the repaired server: its first boot on the new file logged SECUREBOOT: Image APPROVED. If Secure Boot was switched off while troubleshooting, turn it back on (VM Options > Boot Options > Secure Boot) once the machine boots cleanly. Leaving it off is the most common thing we find afterwards.
Verifying the fix
- On the next power-on,
vmware.logshowsDICT nvram = "<vmname>.nvram"and a single power-on, not a run of them. SECUREBOOT: Image APPROVEDappears, if Secure Boot is enabled.- VMware Tools goes to running and stays there.
bcdedit /enum firmwarelists Windows Boot Manager with the correct device.- Restart the VM once more on purpose. A repair you only proved on the first boot is not proven.
- Delete the
.oldand.bakcopies only after the machine has been through a backup cycle on the new file.
The takeaway
When a VM will not boot but its disks are healthy, look one level down at the firmware store before you start restoring disks from backup. The .nvram file is small, easy to regenerate, and easy to point at the wrong place by accident. A misplaced TRUE in a configuration file can quietly swap out a server's boot entries and still let it start. Ten minutes in the VM folder and its log usually tells you which of those happened, and the repair is a rename and a reboot rather than a restore.
If you have a virtual machine that will not come back after maintenance, or a vSphere estate nobody has looked at closely in a while, get in touch or call us at 970-282-8838. This is the kind of work we do every week for the systems we host and manage.