convert/convert_windows.ml: Disable VMware drivers

[rwmjones]

We essentially expect that uninstalling VMware drivers will fail (because the VMware uninstaller is broken). Therefore add a firstboot script which does the next best thing, disabling all VMware drivers.

This requires the Microsoft devcon utility is installed under /usr/share/virt-tools/{x86,x64}/devcon.exe. This utility is open source and can be found here:
https://github.com/microsoft/Windows-driver-samples/tree/main/setup/devcon

Most of the hard work was done by and all of the batch file was written by Vadim Rozenfeld.

Thanks: Vadim Rozenfeld
Fixes: https://issues.redhat.com/browse/RHEL-112249

[crobinso]

Looks ok to me. I'll give it a test

[rwmjones]

I don't think it's going to work at the moment, I spotted at least one bug. (Hence draft!)

[rwmjones]

This still doesn't work, because apparently the devcon.exe utillity is weirdly broken when we run it on Windows.

[rwmjones]

The patch is now in a state where it runs and at least tries to disable the VMware drivers.

However for each VMware driver, the disable command fails. I'm not sure if that's what it is supposed to do or not. No additional information is given by devcon, so it's not clear why it fails to disable the driver and if that's expected or not. I will try to attach the latest firstboot log. (@vrozenfe please take a look)

NB: Before testing you will need to install mingw-devcon-redistributable-0~20260212git2977346e-5.fc45.noarch.rpm (release -5 contains vital fixes)

[rwmjones]

log.txt

[rwmjones]

setupapi.dev.log

[rwmjones]

I reran the same conversion using the MSFT version of devcon.exe, ie. known working. It also failed to disable the devices. The output of devcon is slightly different. Diffing the log from the comment above and the log from the MSFT version of devcon, the difference (for each device, I only show one) is:

 Device: VMware Pointing Device
 ID: @ACPI\VMW0003\4&1BD7F811&0                                   
 ---------------------------------------------------------------
 Disabling device: VMware Pointing Device
-devcon.exe: command disable failed
+ACPI\VMW0003\4&1BD7F811&0                                   : Disable failed
+No matching devices found.

Two conclusions from this:

• It seems as if something about the guest image I am using prevents the drivers from being disabled.
• There could still be an issue with how error messages get printed in the mingw build. (I will investigate this issue separately)

[rwmjones]

The mingw devcon.exe problem was me missing a message catalog so messages were missing from the binary. This will be fixed in the 0-6 Fedora package.

[rwmjones]

@crobinso I believe this is finally ready for testing now. I wasn't able to satisfactorily test that driver disabling works unfortunately, but I did at least prove that the new script runs at the right time and fails in the same way both with MSFT devcon and our devcon, which is ... something?!

Edit: You will need mingw-devcon-redistributable-0~20260212git2977346e-6.fc45.noarch.rpm

[crobinso]

OK I tested it out. Confirmed the code is executing and output is in the windows log. This is a windows 10 VM from vmware. I see the same Disable failed\nNo matching devices found. for every device.

For starters the No matching devices found seems like crappy fallthrough error printing. It happens even if an operation is blocked due to lacking admin privs as shown here: https://stackoverflow.com/a/29251295

Running manually devcon.exe disable fails like that for all the VMWare devices.

Seems like devcon disable is the equivalent of Device Manager -> Right Click -> Disable Device. Find a device in device manager, right click -> Details -> Events will show the device name that you can then pass to devcon disable to confirm that is what it's doing (you'll see it reflected in the device manager UI). THis seems to be like offlining a device and is persistent across reboots (I confirmed for one device).

So it makes sense that devcon disable fails running outside vmware, because no vmware devices show up in device manager.

There is devcon remove which works on vmware devices for me, when run manually (I didn't attempt editing firstboot scripts). I can't tell exactly what this does though. It probably is removing device registry entries, like some persistent mapping from the device PCI/SCSI/ACPI/etc address to what driver it was previoulsy bound to. After running devcon remove, the device doesn't show up in devcon findall and that change is persistent across reboots.

However in the case of the vmci driver, even after devcon remove it still showed up in pnputil /enum-drivers. chatgpt tells me that output lists drivers in the 'windows driver store', but that doesn't include every on disk driver, and says nothing about if they are loaded or not. So I think /enum-drivers is some kind like package manager list for drivers, but isn't exhaustive? No other vmware devices showed up in that for my VM.

If you grab the oem42.inf name out of /enum-drivers output and pass it to pnputil /delete-driver oem42.inf /uninstall /force it actually deletes the driver from Windows\System32\Drivers.

Also it seems like pnputil /enum-devices gives us same but noisier output as devcon.exe findall, and pnputil /remove-device is devcon.exe remove but I didn't fully investigate.

[crobinso]

So to summarize a bit: AFAIK devcon disable isn't expected to work here. There shouldn't be any active vmware devices once we migrate off vmware. if there are that seems like something we should understand better!

devcon remove seems to remove registry evidence of vmware devices that were previously attached. maybe that has some effect? but not clear what value is after moving off vmware

But if success here is devcon findall * | findstr /i "VMWARE" is empty, I think switching devcon disable to devcon remove will do that. Or am I misunderstanding what the goal is here? @vrozenfe @YanVugenfirer

[rwmjones]

I should note that Vadim (privately) gave me two scripts. The one I used for this patch uses devcon disable, the other script is basically the same but using devcon remove. I wanted to stick to the simpler & safer change, but we can "upgrade" to removing the drivers easily later if we decided to do that.

[YanVugenfirer]

OK I tested it out. Confirmed the code is executing and output is in the windows log. This is a windows 10 VM from vmware. I see the same Disable failed\nNo matching devices found. for every device.

For starters the No matching devices found seems like crappy fallthrough error printing. It happens even if an operation is blocked due to lacking admin privs as shown here: https://stackoverflow.com/a/29251295

Running manually devcon.exe disable fails like that for all the VMWare devices.

Seems like devcon disable is the equivalent of Device Manager -> Right Click -> Disable Device. Find a device in device manager, right click -> Details -> Events will show the device name that you can then pass to devcon disable to confirm that is what it's doing (you'll see it reflected in the device manager UI). THis seems to be like offlining a device and is persistent across reboots (I confirmed for one device).

BTW: you can also choose "View->Show Hidden devices" in the device manager. Then you can see devices that are not currently connected and non-pnp devices.
In any case it will only disable the device not remove the drivers.

So it makes sense that devcon disable fails running outside vmware, because no vmware devices show up in device manager.

There is devcon remove which works on vmware devices for me, when run manually (I didn't attempt editing firstboot scripts). I can't tell exactly what this does though. It probably is removing device registry entries, like some persistent mapping from the device PCI/SCSI/ACPI/etc address to what driver it was previoulsy bound to. After running devcon remove, the device doesn't show up in devcon findall and that change is persistent across reboots.

However in the case of the vmci driver, even after devcon remove it still showed up in pnputil /enum-drivers. chatgpt tells me that output lists drivers in the 'windows driver store', but that doesn't include every on disk driver, and says nothing about if they are loaded or not. So I think /enum-drivers is some kind like package manager list for drivers, but isn't exhaustive? No other vmware devices showed up in that for my VM.

devcon little bit outdated. See comment from MS: https://github.com/MicrosoftDocs/windows-driver-docs/blob/staging/windows-driver-docs-pr/devtest/devcon-disable.md

If you grab the oem42.inf name out of /enum-drivers output and pass it to pnputil /delete-driver oem42.inf /uninstall /force it actually deletes the driver from Windows\System32\Drivers.

Also it seems like pnputil /enum-devices gives us same but noisier output as devcon.exe findall, and pnputil /remove-device is devcon.exe remove but I didn't fully investigate.

For general knowledge:
In general MS distinguishing between HW first and SW first driver installation scenarios.
HW first means the HW is present or hot pluged, it is enumerated by the bus driver it is connected to and device manager prompts for driver installation.

SW first means you install your driver into driver store before any might HW exists in the system. What is "driver store" depends on the how old your Windows is (something between not really anything, to the ability to have side by side driver versions in the system). Latest MS certifications enforce drivers' package isolation https://learn.microsoft.com/en-us/windows-hardware/drivers/develop/driver-isolation

[YanVugenfirer]

BTW: I suggest removing any remnant of the VMWare installation completely. Including registry entries and directories.

[rwmjones]

I worry about deleting directories in v2v as someone might put some random important data in them & then blame us.

[YanVugenfirer]

1. Sorry, I missed something in this conversation - let's not use downstream mingw-devcon.

2. Please delete the drivers completely. Disabling is not enough. As we saw during live debugging, phantom NIC with assigned IP prevented setting the same IP to active NIC.

I worry about deleting directories in v2v as someone might put some random important data in them & then blame us.

It is easy to learn what should be the content of the VMWare installation and warn and not delete the content if something else is present. But let's clean up VMWare completely, also to prevent unneeded questions.

@kostyanf14

[rwmjones]

Sorry, I missed something in this conversation - let's not use downstream mingw-devcon.

What does this mean?

Please delete the drivers completely. Disabling is not enough. As we saw during live debugging, phantom NIC with assigned IP prevented setting the same IP to active NIC.

Do you mean running devcon remove instead of devcon disable?

Vadim has a private git repo with some scripts, can we publish that please so I can refer to it. With that repo in mind, let's be specific about exactly what is involved. Deleting "registry entries and directories" is not a sufficient description to implement anything. Which scripts from Vadim's repo need to be run, in what order, with rebooting between or not, etc.

Anyway (as this is upstream v2v discussion) I would prefer to do the safest and smallest change first. We can easily change from remove -> disable later if this works well.

[YanVugenfirer]

Sorry, I missed something in this conversation - let's not use downstream mingw-devcon.

What does this mean?

mingw-devcon-redistributable-0~20260212git2977346e-5.fc45.noarch.rpm
Modern systems include pnputil.

Please delete the drivers completely. Disabling is not enough. As we saw during live debugging, phantom NIC with assigned IP prevented setting the same IP to active NIC.

Do you mean running devcon remove instead of devcon disable?
Yes.

Vadim has a private git repo with some scripts, can we publish that please so I can refer to it. With that repo in mind, let's be specific about exactly what is involved. Deleting "registry entries and directories" is not a sufficient description to implement anything. Which scripts from Vadim's repo need to be run, in what order, with rebooting between or not, etc.

Anyway (as this is upstream v2v discussion) I would prefer to do the safest and smallest change first. We can easily change from remove -> disable later if this works well.

Let's start with Vadim scripts. But little understanding of how VMWare installation looks like is in oder in my opinion. If you need help with this, let's discuss.

[rwmjones]

I'm going to be very straightforward and open about my position here, so there's no confusion.

I do not believe that "leftover VMware drivers" is an actual problem, because I have not seen incontrovertible evidence of that. If that evidence is later produced, I will change my position on this.

Virt-v2v converts literally 1000s of VMs every day. A single mistake that lasts for a few days in production is a catastrophe because it would mean having to go in after the fact to angry customers to clean up problems (if that is even possible, accidentally deleting vital data or registry keys isn't recoverable). This makes me very conservative about making changes. We should do the smallest thing that can work.

Everything we ship has to be open source, since this is an open source project. We have to use tools that are either installed in all VMs we support, or open source tools we can compile ourselves. I wasn't aware until this point that we shoudn't be using devcon and should use pnputil because no one bothered to tell me, and all the scripts provided to me so far use devcon. (Side question: pnputil appeared first in Windows Vista which is old enough for us, but does that version have all the features we need?) Because I was told to use devcon, I have packaged devcon using an open source toolchain and got it into Fedora. This was a significant amount of work in itself.

I do have a rough idea of what VMware installs in Windows. I don't know (or at least, not without a lot of extra effort) precisely what registry keys it adds, the complete list of files it adds, and whether those lists have changed over time, as I imagine they have. To thoroughly remove VMware by hand we'd presumably need to do that detailed study. This sounds to me far too complex and hence risky (see above).

The place to discuss all this is right here, on the open source upstream project.

Vadim's repo should be publicly available unless there is some clear reason why it should not be, and I don't see that reason.

[kostyanf14]

I wasn't aware until this point that we shoudn't be using devcon and should use pnputil because no one bothered to tell me, and all the scripts provided to me so far use devcon.

MSFT currently recommends using PnPUtil instead of DevCon, see https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/devcon. So, as we develop a new feature, the best solution goes with MSFT recommendation.

I do not believe that "leftover VMware drivers" is an actual problem

At least keeping the network configuration is a problem. Windows is very strict when check network configuration. If you try to set the same IP + mask that are in use by any other adapter (even if it is disabled), Windows will show error and ask you to remove configuration. Exact this issue we caught in customer environment when scripts failed to set IP for NetKVM because IP was in use by vmxnet3. There was no vmxnet3 device, but driver was present and Windows keeping configuration for future.

The similar issue we will see with vsock. In Windows you need to register socket user mode and address family. Windows does not have virtio-sock AF, so VMWare vsock register AF and link AF with corresponding DLL. As a result, installing virtio-win viosock user mode driver will fail, as the same AF is already registered. We don't see this issue yet, only because this is new driver that was release official only in the latest version of virtio-win.

[YanVugenfirer]

I will start from the end.

Vadim's repo should be publicly available unless there is some clear reason why it should not be, and I don't see that reason.

Totally agree. I don't see any reason why it shouldn't be publicly available.

I'm going to be very straightforward and open about my position here, so there's no confusion.

I do not believe that "leftover VMware drivers" is an actual problem, because I have not seen incontrovertible evidence of that. If that evidence is later produced, I will change my position on this.

1. We had at least 1 BSOD with VMWare driver in the stack
2. And yet those are distractions from working on additional root causes. Both because customer is asking and needs answers, and because we waste our time during crash dump debugging.
3. Just a week ago we had pointless discussion with customer during what they perceived as high priority crisis with bunch of people from RH during recharge day and customer insisting to understand why there are still VMWare leftovers. Let's prevent this from happening in the future.

Virt-v2v converts literally 1000s of VMs every day. A single mistake that lasts for a few days in production is a catastrophe because it would mean having to go in after the fact to angry customers to clean up problems (if that is
even possible, accidentally deleting vital data or registry keys isn't recoverable). This makes me very conservative about making changes. We should do the smallest thing that can work.

Perfectly good reason to do things on Windows the Windows way.

Everything we ship has to be open source, since this is an open source project. We have to use tools that are either installed in all VMs we support, or open source tools we can compile ourselves. I wasn't aware until this point that we shoudn't be using devcon and should use pnputil because no one bothered to tell me, and all the scripts provided to me so far use devcon. (Side question: pnputil appeared first in Windows Vista which is old enough for us, but does that version have all the features we need?) Because I was told to use devcon, I have packaged devcon using an open source toolchain and got it into Fedora. This was a significant amount of work in itself.

@kostyanf14 and I are more than happy welcomed to participate in research and discussions to prevent inventions of the wheel if there are licensing, redistribution, or other issues.

I do have a rough idea of what VMware installs in Windows. I don't know (or at least, not without a lot of extra effort) precisely what registry keys it adds, the complete list of files it adds, and whether those lists have changed over time, as I imagine they have. To thoroughly remove VMware by hand we'd presumably need to do that detailed study. This sounds to me far too complex and hence risky (see above).

Is this an effort? Yes.
But will low risk, as we should do what VMWare installer does - so the risk is on the same level.

The place to discuss all this is right here, on the open source upstream project.

Yes, but getting additional help to solve Windows issues (like research of VMWare installer) is a resource allocation question that can be discussed internally.

[crobinso]

Those cases about vmxnet and vsock are helpful, thanks.

There was also a case where vmware drivers were in a kernel backtrace, right? Do we know how that happened, when VM was running outside of vmware?

Just throwing it out there: IMO a standalone vmware uninstaller exe/msi would be useful. Not sure if yall have seen this gist but there is a very active discussion of people interested in this problem: https://gist.github.com/broestls/f872872a00acee2fca02017160840624 . It suggests to me though that this problem space is kinda too elaborate to live embedded in virt-v2v code (and also maybe we can get contributors to polish off corner cases). If a tool was shipped in RH's virtio-win package, virt-v2v could upload it to the VM and run it on firstboot.

[YanVugenfirer]

@crobinso We also have a tested code as part of our installer in DLL that has the functionality of driver installation and removal that can be wrapped and used for generic purposes.

[rwmjones]

Question: If we disable a driver (using devcon disable or whatever the equivalent pnputil command is), and then reboot the system, is it expected that Windows will never load that driver into memory?

And will that also fix the network adapter address / vsock address problems?

[rwmjones]

Not sure if yall have seen this gist but there is a very active discussion of people interested in this problem: [...]

Yes, we link to this, plus the VMware official method, in our docs: https://libguestfs.org/virt-v2v.1.html#removing-vmware-tools-from-windows-guests

My view is that doing either of these is just too complex. Even if we try to "shift the blame" to someone else, we will inevitably be asked to support / fix the resulting mess. So we suggest to users that if they are really concerned about the "leftover VMware drivers" "problem" they should read those methods and if they are happy, then they can do it themselves.

[kostyanf14]

Question: If we disable a driver (using devcon disable or whatever the equivalent pnputil command is), and then reboot the system, is it expected that Windows will never load that driver into memory?

And will that also fix the network adapter address / vsock address problems?

No. Windows keeps the network adapter address until the driver is installed or the adapter address is removed from the registry. Windows keeps any custom address family until you unregister it via API or remove it from the registry.

[rwmjones]

No. Windows keeps the network adapter address until the driver is installed or the adapter address is removed from the registry. Windows keeps any custom address family until you unregister it via API or remove it from the registry.

I should note that MTV adds an extra firstboot script which does some network stuff on top of what virt-v2v does. It's probably best to read the scripts to understand exactly what: https://github.com/kubev2v/forklift/tree/main/pkg/virt-v2v/customize/scripts/windows

[YanVugenfirer]

@rwmjones The mess is now. In practice we are trying to shift the blame by sending our customers to do some work that they don't do. And latter they complain to us anyway.
It is time to shift the mindset, we are responsible for the whole process and making it easy and painless to the customer.
Yes - we will need to support VMWare removal tool, if we cannot educate the customer to use other methods.

[rwmjones]

Can someone answer this question please:

Question: If we disable a driver (using devcon disable or whatever the equivalent pnputil command is), and then reboot the system, is it expected that Windows will never load that driver into memory?

[YanVugenfirer]

Depending what kind of driver.

• In general case. No. The windows of course can load the driver for the device that was disabled if it will detect another device of the same type or different instance of the same device (here we dive into if PNP ID for specific HW has serial number like USB or not). The disable only for the specific instance.
  Filter drivers are not "disabled" per-se. Should be removed from the stack.

• In our case, maybe (because "HW" will nor be present). But I don't know what kind of driver is VMWare balloon driver, and as Konstantin mentioned for AF family registration will not be affected.

Anyway, please please uninstall the drivers!!!!

[rwmjones]

Anyway, please please uninstall the drivers!!!!

We're waiting on actionable instructions on how to do this. Also for Vadim's repo to be published somewhere, assuming that will still be part of this.

[rwmjones]

In general case. No. The windows of course can load the driver for the device that was disabled if it will detect another device of the same type or different instance of the same device (here we dive into if PNP ID for specific HW has serial number like USB or not). The disable only for the specific instance.

I don't get what disabling a device does if Windows can decide to reenable it. Anyway we are not presenting any VMware devices to qemu guests, so this seems like a non-issue.

Filter drivers are not "disabled" per-se. Should be removed from the stack.

Are there actual VMware filter drivers involved here (be specific, what is the name of the driver?)

In our case, maybe (because "HW" will nor be present). But I don't know what kind of driver is VMWare balloon driver, and as Konstantin mentioned for AF family registration will not be affected.

What does "AF family registration" mean? AF = Address Family? Anyway we aren't presenting a VMware balloon device so I don't know how this could be relevant to anything.

[rwmjones]

So far we've identified exactly two real problems to my knowledge:

• The Windows registry remembers network addresses associated with some VMware network and vsock drivers. Solution here seems like running Remove-NetIPAddress commands.
• Some Windows scanning software flags up VMware drivers which eventually become out of date (because we're not on VMware and so the drivers are no longer being automatically updated). Unclear if the current patch, but replacing devcon disable with devcon remove would clear these up.

[YanVugenfirer]

https://github.com/virtio-win/vmwremover.git

[YanVugenfirer]

In general case. No. The windows of course can load the driver for the device that was disabled if it will detect another device of the same type or different instance of the same device (here we dive into if PNP ID for specific HW has serial number like USB or not). The disable only for the specific instance.

I don't get what disabling a device does if Windows can decide to reenable it. Anyway we are not presenting any VMware devices to qemu guests, so this seems like a non-issue.

Disable = disable specific instance of the device. Windows disables SW representation of the device, not the driver.
I will post screenshots for more clarity soon.

Filter drivers are not "disabled" per-se. Should be removed from the stack.

Are there actual VMware filter drivers involved here (be specific, what is the name of the driver?)

In our case, maybe (because "HW" will nor be present). But I don't know what kind of driver is VMWare balloon driver, and as Konstantin mentioned for AF family registration will not be affected.

What does "AF family registration" mean? AF = Address Family? Anyway we aren't presenting a VMware balloon device so I don't know how this could be relevant to anything.

While we add those, for now let's delete the drivers the we know of.

[YanVugenfirer]

3 NetKVM devices and 1 e1000e

Let's disable the device

One device is disabled, two others are enabled

[YanVugenfirer]

https://github.com/virtio-win/vmwremover.git

@rwmjones https://github.com/virtio-win/vmwremover/blob/main/drivers/remove_vmware_driver_packages.bat - enumerates everything with pnputil (including filter drivers) and removes them

No need for devcon redistribution or clone.

[YanVugenfirer]

@crobinso I understand from Vadim that some registry issues you found are not in https://github.com/virtio-win/vmwremover.git, please send PR.

[rwmjones]

https://github.com/virtio-win/vmwremover.git

@rwmjones https://github.com/virtio-win/vmwremover/blob/main/drivers/remove_vmware_driver_packages.bat - enumerates everything with pnputil (including filter drivers) and removes them

Finally we get something actionable, thank you.

No need for devcon redistribution or clone.

I see this script has been rewritten to use pnputil instead of devcon as was in the earlier version that was provided privately.

I'm going to close this PR since we will need to start again from scratch.

[rwmjones]

Please continue discussions here: #144