A Windows Server virtual machine goes through its overnight updates and restart. In the morning it shows as powered on in vCenter, but nothing answers: no ping, no file shares, no Remote Desktop, and VMware Tools reports that it is not running. Open the console and you find the Windows Boot Manager error screen:
Windows failed to start. A recent hardware or software change might be the cause.
File: vpci.sys
Status: 0xc0000098
Info: The operating system couldn't be loaded because a required file is missing or contains errors.We hit exactly this on a production Windows Server 2025 VM running on ESXi 8. Our first guess was the VM's firmware store, its .nvram file, because we had repaired a corrupt one on another server a few weeks earlier. It was not the NVRAM. This is how we narrowed it down and got the server back without restoring it from backup.
TL;DR: Error 0xc0000098 onvpci.sysmeans the Windows boot loader opened the Hyper-V Virtual PCI driver and could not use it. The VM is not hung; it is sitting on an error screen waiting for a key. On a VMware VM nothing uses that driver, so the fast, safe fix is to boot the Windows install ISO, load the offline SYSTEM registry hive and change thevpciservice'sStartvalue from 0 to 4 (disabled). Our server booted on the next try and finished rolling back the failed update. Do not do this on Hyper-V or Azure, where the driver can be in use.
The boot error was only the last link in the chain. The update failed because of a leftover event log registration that had quietly blocked every cumulative update on that server for over a year, and once it was removed, Windows Update still reported a "successful" install that never happened. Both are covered below, along with how we finally got the update in.
What the error actually means
vpci.sys is the Microsoft Hyper-V Virtual PCI Bus driver. Every Windows Server install has it, and it is registered as a boot-start driver, so the boot loader (winload.efi) loads it on every boot whether or not the machine runs on Hyper-V. On Hyper-V it carries hardware that is passed through to the virtual machine. On a VMware VM nothing is attached to it.
Status 0xc0000098 is STATUS_FILE_INVALID. The boot loader found the file and opened it, then decided it could not use it. Because a boot-start driver failed, the loader stops and shows the Boot Manager screen instead of handing control to the Windows kernel.
Why it looks like a hang: that Boot Manager screen waits for someone to press Enter. Monitoring sees a powered-on VM with no heartbeat and no network, and it stays that way until someone opens the console. If your alerting only says "server down", open the console before you assume anything.
Step 1: Find out which stage of the boot is failing
The VM's own log shows where the boot stops before you change anything. ESXi keeps it in the VM's folder on the datastore as vmware.log. A power-off and power-on starts a new log and keeps the old ones as numbered vmware-N.log files; a restart from inside Windows stays in the same log. From the ESXi shell, or on a copy downloaded from the datastore browser:
grep -E 'About to do EFI boot|transitioned to runtime|guest has requested|Triple fault' vmware.logOn a healthy boot the firmware starts Windows Boot Manager and, a few seconds later, the boot loader hands over to the kernel (log lines trimmed for width):
11:37:57 Chipset: The guest has requested that the virtual machine be hard reset.
11:37:59 Guest: About to do EFI boot: Windows Boot Manager
11:38:05 Guest: Firmware has transitioned to runtime.On every failed boot the last line never came:
12:06:40 Chipset: The guest has requested that the virtual machine be hard reset.
12:06:42 Guest: About to do EFI boot: Windows Boot ManagerThat one comparison rules out a lot. The firmware ran and started Windows Boot Manager from the EFI system partition, so the virtual hardware, the NVRAM and the boot entry were all doing their jobs. The failure is inside the Windows boot loader, after it starts and before the kernel runs.
The log also showed something we would not have guessed. The scheduled restart at 4:37 AM booted cleanly, and Windows went on to finish installing the update. Then, at 5:06 AM, the server restarted again on its own. That second restart is the one that never came back. As we found out later (see below), it was Windows rolling back an update that had failed.
Gotcha: the restart that breaks a server is not always the one you scheduled. Read the log for every reset that night, not just the one at your maintenance time. We also saw VMware's "virtual CPU has entered the shutdown state" (triple fault) message once during the hands-on attempts. It sounds alarming, but it came out of the stuck boot and did not return after the fix.
Step 2: Get to a command prompt
Attach the Windows Server install ISO to the VM's CD drive. Press Esc at the Boot Manager error screen to reach the VMware boot menu, choose the CD drive, and press a key when asked to boot from CD. In Setup pick your language and keyboard, then Repair my PC (not Install), then Troubleshoot and Command Prompt. The Windows Server recovery menu has no "Uninstall updates" option, so everything below happens at the prompt.
The recovery environment often gives the Windows volume a different drive letter, so find it first:
for %d in (C D E F G) do @if exist %d:\Windows\System32\ntoskrnl.exe echo Windows is on %d:The examples below assume C:.
Gotcha: the recovery environment runs on Pacific time by default, so file times from the server's disk can be off from your local time by an hour or more. Convert before you match a file to an event in the log.
Step 3: Check whether an update was left half-finished
dir C:\Windows\WinSxS\pending.xmlIf pending.xml exists, Windows Update has queued changes it has not finished applying. The standard move is to cancel them:
mkdir C:\Scratch
dism /image:C:\ /scratchdir:C:\Scratch /cleanup-image /revertpendingactionsPoint /scratchdir at the Windows disk. The recovery environment's own scratch space is a small RAM disk, and DISM warns that it may be too small.
In our case DISM failed with 0x8000ffff every time, from the recovery media and from the server's own copy of DISM. The log at X:\Windows\Logs\DISM\dism.log explained why:
CBS Exec: Poq is not partially committed, rollback everything
CSI Cancel called with rollback transaction pending
CBS Failed to execute reverse operations.: E_UNEXPECTEDWindows was already in the middle of rolling the update back and had queued the rest of that rollback, so there was nothing for DISM to cancel. In other words, /revertpendingactions cannot help when the pending work is itself a rollback. The fix has to get the machine booting so Windows can finish the job.
Step 4: Rule out the usual suspects
Before changing anything, we checked the things that most often cause a boot loader error. All of them came back clean:
- The file itself.
dir C:\Windows\System32\drivers\vpci.sysshowed it present at its normal size, anddir /s C:\Windows\System32\DriverStore\FileRepository\vpci.sysfound a matching copy in the driver store. - Where Windows loads it from. Loading the offline registry (see Step 5) and querying
Services\vpcishowed the normalSystem32\drivers\vpci.syspath. - The disk.
chkdsk C: /ffound no problems and no bad sectors. - Boot configuration.
mountvol S: /sfollowed bybcdedit /store S:\EFI\Microsoft\Boot\BCD /enumshowed no debugging, hypervisor or safe mode settings left behind. - NVRAM. The VM's configuration pointed at the right
.nvramfile and it was the normal size, which matches what the log had already told us.
fsutil file layout C:\Windows\System32\drivers\vpci.sys is worth a look too. It lists the file's streams, extended attributes and fragments. Ours carried an antivirus scan-cache attribute, but drivers that loaded fine carried the same attribute, so it was not the cause.
Step 5: Disable vpci.sys offline
Only do this on a VMware VM. On Hyper-V and Azure, vpci.sys can carry real devices. And if the file named on your error screen is a storage or file system driver, disabling it leaves Windows unable to find its own disk; restore the file or the server from backup instead.
A driver's Start value in the registry tells Windows when to load it: 0 means the boot loader loads it, 4 means it is disabled. Changing vpci from 0 to 4 stops the boot loader from touching the file at all. From the recovery command prompt:
reg load HKLM\OFFSYS C:\Windows\System32\config\SYSTEM
reg query HKLM\OFFSYS\Select /v Current
reg query HKLM\OFFSYS\ControlSet001\Services\vpci /v Start
reg add HKLM\OFFSYS\ControlSet001\Services\vpci /v Start /t REG_DWORD /d 4 /f
reg unload HKLM\OFFSYSSelect\Currenttells you which control set Windows boots from. It is almost always 1, which isControlSet001. If it shows 2, useControlSet002.- Write down the old
Startvalue (ours was 0) so the change can be reversed. - Always run
reg unloadbefore rebooting, so the change is written back to the file on disk.
If you prefer a GUI, regedit runs in the recovery environment too. Select HKEY_LOCAL_MACHINE, choose File, Load Hive, open the same SYSTEM file and edit the same value, then use File, Unload Hive when you are done.
What happened next
We ran wpeutil reboot, let the VM boot from its own disk and watched the log. The boot loader handed over to the kernel 12 seconds after Boot Manager started, the VMware drivers loaded, and VMware Tools reported in about two minutes later. Shortly after that the server answered ping and file sharing.
The console then showed Windows' "Something didn't go as planned" screen and said it was undoing changes. That is Windows finishing the rollback it had started before the failed restart. Because that first attempt was cut off, it starts the rollback over from the beginning, so it takes longer than the original install did. Leave it alone until it finishes, and expect at least one more restart.
Watching the rollback: the screen shows no percentage, butC:\Windows\Logs\CBS\CBS.logdoes show progress. The server accepts file sharing connections during this stage, so you can read the log over the admin share (\\server\C$) from another machine with an administrator login. Look for theBegin executing advanced installerandProcessing Namespacelines, and forRunning poqexec, which is the final file swap before the restart.
What CBS.log showed afterwards
Once the server was reachable, CBS.log filled in the timeline:
- At the 4:31 AM shutdown the cumulative update installed, including a newer version of the Hyper-V Virtual PCI driver package (
wvpci.inf). - After the 4:37 AM restart it ran its configuration steps up to 100%, then one of the last ones failed (
eventsinstaller.dll, error80073aa2). Windows logged0x800f0922(CBS_E_INSTALLERS_FAILED) and started rolling the whole update back. - At 4:56 AM the rollback reinstalled the previous Virtual PCI driver package, then swapped files and restarted at 5:06 AM.
- That restart failed on
vpci.sys.
We never proved exactly why the boot loader rejected that copy of the file. The disk was clean and the file looked normal, but the timing points squarely at the rollback's driver swap, ten minutes before the failed boot. Disabling a driver the platform never uses is a sound fix either way. The more important question was why the update failed in the first place.
The real cause: a leftover event log registration
Error 80073aa2 is HRESULT_FROM_WIN32(15010). A few lines above it, CBS.log said the update's event log manifest could not be registered because one of its names was already declared by an existing provider. Windows keeps a registration for every event log publisher under:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{publisher GUID}Each one points at a message DLL. On our server one of them, the publisher for the Microsoft Store client user interface, still pointed at a DLL that no longer existed. It was a leftover from a component removed long ago, and every cumulative update that tried to register the current version of that publisher collided with it, failed at the very end and rolled back.
The component store showed how long this had been going on. Every monthly cumulative update for more than a year was staged but none was installed; the server had been sitting on an older build the whole time, while patch reports showed it as up to date. This PowerShell finds publishers whose message DLL is missing:
Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publishers' |
ForEach-Object { $p = Get-ItemProperty $_.PSPath
$dll = [Environment]::ExpandEnvironmentVariables([string]$p.MessageFileName)
if ($dll -and -not (Test-Path $dll)) { '{0} {1} {2}' -f $_.PSChildName, $p.'(default)', $dll } }Treat the output as a list of suspects, not a delete list: only the publisher that CBS.log names as the conflict matters. We exported that one key to a .reg file as a backup, deleted it, and moved on to the update.
Gotcha: "staged" is not "installed".DISM /Online /Get-Packagesor theCurrentStatevalues underHKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packagestell you which.0x70is installed,0x40is only staged. A server whose newest rollup is staged and never installed is not patched, whatever the update history says.
Windows Update said "Succeeded". It was not.
With the conflict gone we installed the October cumulative update through Windows Update. It reported success in six seconds and asked for a restart. The server restarted quickly and came back on the same old build, and Windows Update offered nothing new. No cumulative update installs in six seconds.
CBS.log showed what happened. Windows Update's update agent re-read the result of the earlier failed attempt, which was marked complete, reported success, and never handed the package to the servicing stack; at shutdown CBS logged Shtd: Nothing is pending. The update was still sitting there staged.
The way around it was to install the staged package directly with DISM, which reports a real result. A staged package keeps its manifest under C:\Windows\servicing\Packages, and DISM accepts that manifest as the package path:
dism /Online /Add-Package /PackagePath:C:\Windows\servicing\Packages\Package_for_RollupFix~31bf3856ad364e35~amd64~~26100.33650.1.28.mum /NoRestartUse the version that is staged on your server. If the update is not staged but Windows Update has downloaded it, the .cab in C:\Windows\SoftwareDistribution\Download works the same way; check its digital signature first. DISM took about forty minutes on this server's slow storage, finished with exit code 3010 (success, restart required), and the restart took another half hour: the shutdown phase, the boot phase where the earlier attempt had failed, and one automatic final restart. This time CBS.log ended with Startup: Processing complete and S_OK, the rollup package showed as installed, the build number had moved up, and Windows Update had nothing left to offer.
Gotcha: never take "Succeeded" on a cumulative update at face value. Check the build number (CurrentBuildandUBRunderHKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion) before and after. We found a second server the same day, an older Windows Server 2019 machine, where Windows Update had reported that month's cumulative update as installed three times without the build ever changing; installing the downloaded.cabwith DISM was what finally moved it.
Loose ends worth checking on the same visit
- Component store health. A full
DISM /Online /Cleanup-Image /RestoreHealthscan found exactly one damaged file, and it sat in an old, superseded version of a component. Newer copies of the same file elsewhere in the store were intact. The repair tried to download a clean copy of that old version from Windows Update and stalled, so the better tool wasDISM /Online /Cleanup-Image /StartComponentCleanup, which removes superseded component versions (and, after a year of stacked updates, a good deal of space with them). Follow it withDISM /Online /Cleanup-Image /ScanHealthandsfc /scannowto confirm. On slow storage, budget an hour or more for each full scan. - The clock. After the cold boots the server came up seven hours fast. ESXi keeps the virtual hardware clock in UTC while Windows reads it as local time, and a server outside a domain may only sync its clock once a week. Point Windows Time at a reliable source with a short polling interval, and tell VMware Tools that it may set the clock backwards at startup (
time.synchronize.tools.startup.backward = TRUEin the VM's advanced settings). - Secure Boot. It had been switched off on this VM during an earlier repair and never switched back on. Before turning it on, compare the signature on
\EFI\Microsoft\Boot\bootmgfw.efiwith the certificates in the VM's firmware (Get-SecureBootUEFI -Name db), or with a VM on the same build that already boots with Secure Boot. If the VM ever refuses to boot afterwards, clearing the checkbox again puts it straight back. - The recovery environment.
reagentc /infoshowed WinRE disabled, which also makes Windows' monthly recovery environment updates fail quietly. The image was still on the recovery partition; only the boot entry pointing at it was missing, most likely lost when the boot configuration was rebuilt during an earlier repair. Give the partition a drive letter, runreagentc /setreimage /path R:\Recovery\WindowsRE /target C:\Windowsand thenreagentc /enable. - The install ISO. Detach it from the VM so it is not part of the next boot.
- The registry change. Write it down. A disabled
vpciservice looks like a mistake to the next person who finds it. If the VM is ever moved to Hyper-V or Azure, setStartback to 0 first. - A known-good backup. Keep one within reach before any of this. Ours was the fallback if the fix had not worked, and a restore is the right first step when the error names a driver the server actually needs.
The takeaway
Read the Boot Manager screen exactly. The file name and status code are the diagnosis; "stuck after updates" is not. Use the VM's log to see which stage of the boot failed before you touch anything, and when the file named on screen is a driver your platform does not use, disabling it offline takes a few minutes and saves a restore.
Then keep going until you know why the update failed. Ours had been failing every month for over a year behind patch reports that said everything was fine. The boot error was the first symptom anyone actually saw. Verify patches by build number, not by "Succeeded".
If a server will not come back after patching, or you want your update windows watched by people who read the logs, get in touch or call us at 970-282-8838. We manage and host Windows servers and VMware environments every day.