Virtual machines appear as invalid or orphaned in vCenter Server

-
Symptoms
  • Virtual Machines show as invalid or orphaned in vCenter Server
  • Virtual Machines show as invalid or orphaned after a VMware High Availability (VMware HA) host failure occurs
  • Virtual Machines show as invalid or orphaned after an ESX host comes out of maintenance mode
  • Virtual Machines show as invalid or orphaned after a failed DRS migration
  • Virtual Machines show as invalid or orphaned after a storage failure
  • Virtual Machines show as invalid or orphaned after the connection is lost between the vCenter Server and the host where the virtual machine resides
  • You see one or more of these errors when trying to start a virtual machine:

    • Could not power VM, no swap file, failed to power on VM.
    • VMControl error -11: No such virtual machine.
    • A general system error occurred. The system returned on error. Communication with the virtual machine may have been interrupted
  • In C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log file, you see entries similar to:

    [11916 error 'vpxdvpxdVmomi'] [SoapInitWritelocked] GetVpxaInfo failed for host esxi01.domain.com
    [11916 info 'vpxdvpxdVmomi' opID=HB-host-01@3777-1a2b429b] [SoapInitWritelocked] Attempting to get Vpxa version for host esxi01.domain.com
    [10060 warning 'Default' opID=HB-host-01@3777-1a2b429b] Closing Response processing in unexpected state: 3
    [11916 info 'vpxdvpxdHostAccess'] [VpxdHostAccess] Failed to get VpxaCnxInfo over SOAP version vpxapi.version.version1 for host esxi01.domain.com:Invalid response code: 503 Service Unavailable

    Note: The preceding log excerpt are only examples. Date, time, and environmental varibales may vary depending on your environment.
Purpose
This article explains what orphaned virtual machines are, how they occur, and how you can fix them. The article outlines the most common errors that relate to orphaned virtual machines and how these issues can be resolved.
Resolution
Notes:
In vCenter Server, you may find that you have a virtual machine that has an orphan designation or has become invalid. An orphan virtual machine is one that exists in the vCenter Server database but is no longer present on the ESX host. A virtual machine also shows as orphaned if it exists on a different ESX host than the ESX host expected by vCenter Server.

A virtual machine can show up as invalid or orphaned in these situations. To resolve these issues, see the troubleshooting steps provided for each situation:

After a vMotion or VMware DRS migration

  1. Connect to the source and destination ESX/ESXi hosts using SSH. For more information, see Opening a command or shell prompt (1003892).
  2. Using the vmware-cmd -l command, check if the orphaned virtual machine is registered on the same ESX host as reported by vCenter Server, which is likely the source machine. If the virtual machine is not registered on that host, use the vmware-cmd -l command to check if it is registered on the destination ESX host.

    Note: In ESXi, use the vim-cmd vmsvc/getallvms command instead of the vmware-cmd -l command.
  3. If the virtual machine is registered on the destination ESX/ESXi host: 

    1. Run these commands to restart the vpxa and ESX host management services:

      service mgmt-vmware restart

      For more information, see Virtual machines unexpectedly reboot after issuing the "mgmt-vmware restart" command (7301769).

      service vmware-vpxa restart

      Note: In ESXi, use the /sbin/services.sh command to restart the management services on the host.
    2. Restart the vCenter Server Server Service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
Notes:
  • For the majority of problems related to orphaned virtual machines on ESX/ESXi, these steps resolve the issue.
  • Ensure that there is no time difference between the source and destination ESX/ESXi hosts.
After completing these steps, you may see these errors when attempting to start your virtual machine:
  • A general system error occurred. The system returned on error. Communication with the virtual machine may have been interrupted.
  • VMControl error -11: No such virtual machine.
Follow these steps to resolve the errors:
  1. Try to register the virtual machine using the vmware-cmd -s command.

    Note: If the command does not fail with the VMControl error -11: No such virtual machine error, go to step 5.

    You can also try to register the virtual machine by right-clicking on its .vmx file in the datastore browser and choosing Register Guest.

    For more information, see Registering or adding a virtual machine to the inventory on vCenter Server or on an ESX/ESXi host (1006160).
  2. Try to start the virtual machine from the command line using:

    vmware-cmd config_path start

    Where config_path is the path to the configuration file as determined by vmware-cmd –l.

    For more information about powering on a virtual machine on an ESXi host using the command line, see Powering on an ESX/ESXi host's virtual machine (1003738).
  3. View the virtual machine's .vmx file and verify that the file has valid configuration parameters. Also ensure that the file contains only UTF-8 characters.

    Note: If possible, compare the .vmx file with the .vmx file of another virtual machine.
  4. Create a new virtual machine using the virtual disks of the original virtual machine.
  5. Power on the new virtual machine.

After a VMware HA host failure occurs, or after the ESX host comes out of maintenance mode

  1. Remove the ESX host from vCenter Server:

    1. Select the specified ESX/ESXi host in the vCenter Server inventory.
    2. Right-click on the ESX/ESXi host and select Disconnect.
    3. After the ESX/ESXi host is disconnected, right-click the host again and select Remove.
  2. Remove the vCenter Server agent and VMware HA agents by running these commands from the service console of the ESX host:

    export LGTO_AAM_VMWARE_REMOVAL=1
    rpm -e LGTOaama
    rpm -e LGTOaamvm
    rpm -e VMware-vpxa
  3. Re-add the ESX/ESXi host to vCenter Server:

    1. Select the cluster, datacenter, or farm to which you want to add the ESX/ESXi host.
    2. Right-click the cluster, datacenter, or farm and choose Add.
    3. Enter the ESX/ESXi host's IP address. You must have a username and password with sufficient permissions to add the host.
    4. Complete the steps in the Add Host wizard.
If you see this error when trying to start the virtual machines:
  • Could not power VM, no swap file, failed to power on VM when trying to start them
Perform these steps:
  1. Run the ps -auxwww | grep -i vm_name command on each ESX host until you find the server host that is running the virtual machine's process and locking its files.

    Note: For ESXi the command is:

    ps | grep vmx
  2. Run the service vmware-vpxa restart command on that ESX host.
  3. Restart the VMware VirtualCenter Server service.
If the issue persists after performing these steps:
  1. Power off the virtual machine.
  2. Access the ESX/ESXi service console using an SSH client.
  3. Open the virtual machine's configuration file (.vmx) using a text editor. The default location is:

    /vmfs/volumes/datastore_name/vm_name/vm_name.vmx
  4. Remove the location of the swap file referenced in the configuration file. It looks similar to:

    sched.swap.derivedName = ""
  5. Save the file.
  6. Rename or delete the existing swap file from the virtual machine directory.
  7. Unregister the virtual machine and re-register it for the changes to take effect. For more information, see Registering or adding a virtual machine to the inventory on vCenter Server or on an ESX/ESXi host (1006160).
  8. Power on the virtual machine.

A virtual machine is deleted outside of vCenter Server


A user can delete a virtual machine through the VMware Management Interface while vCenter Server is down, through the vSphere Client directly connected to an ESX/ESXi host, or by deleting the virtual machine's configuration file through the service console. These virtual machines can be removed from the vCenter Server by right-clicking the virtual machine and selecting delete. If the delete option is not available, see Deleting an orphaned virtual machine when the Remove option is not available (1011468).

Note: If the configuration file was deleted and the virtual disk remains, you can recreate the virtual machine using the VMware Management Interface or the vSphere Client and choosing to attach the existing virtual disk to a newly created .vmx file.

vCenter Server is restarted while a migration is in progress

This is a temporary situation. During startup, vCenter Server reconnects to all hosts. If a migration is completed while vCenter Server is down, a virtual machine can be reported as an orphan until vCenter Server establishes a connection to the virtual machine's target host.

Too many virtual machines are scheduled to be relocated at the same time


An ODBC timeout can cause errors in the database. To resolve an ODBC timeout:
  1. Delete the orphans after ensuring the original virtual machines can power on.
  2. Click DeleteVM on the vCenter Server console to delete the virtual machine and its orphan from the vCenter Server database.

Attempting to delete virtual machines when an ESX/ESXi host local disk (particularly the root partition) has become full

The orphaned virtual machine must be removed by right-clicking the virtual machine in the Inventory.

For more information on correcting the host, see Investigating disk space on an ESX or ESXi host (1003564).

Rebooting the host within 1 hour of moving or powering on virtual machines

If the host is rebooted within an hour after the virtual machines are moved or powered on, the vmInventory.xml file does not persist and hostd does not report these virtual machines as registered on the host. In an HA-enabled cluster, this condition impacts only the virtual machines with HA restart disabled.

For more information, see Virtual machines are orphaned after rebooting the ESXi host on which they reside (2013301).

A .vmx file contains special characters or incomplete line item entries

Using a text viewer, view the contents of the .vmx file. If any lines appear incomplete or contain trailing spaces or special characters:
  1. Make a backup copy of the .vmx file.
  2. Using a text editor, delete the partial lines and/or remove any trailing spaces or special characters.
  3. In vCenter, right-click on the virtual machine and click Remove From Inventory.
  4. In vCenter Server, browse the datastore and find the .vmx for the virtual machine. 
  5. Right-click on the virtual machine and click Add to Inventory.
After completing these steps, the virtual machine shows normally. If you attempt to power it on and receive a "virtual machine question" prompt, respond with the default and the virtual machine will power on normally.

Reload all invalid virtual machines on a single host at one time

This script reloads all invalid virtual machines on a single host at once:
for a in $(vim-cmd vmsvc/getallvms 2>&1 |grep invalid |awk '{print $4}'|cut -d \' -f2);do vim-cmd vmsvc/reload $a;done
vCenter database is full
To resolve this issue, manually purge or truncate the vCenter Server database.Purging old data from the database used by vCenter Server (1025914)

Comments

Post a Comment

Popular posts from this blog

esxi-host-shows-disconnected-vmware

-
Symptom: The host and VMs in a vcenter server appears to be disconnected. Issue: The host and its corresponding vms could appear as disconnected due to issue with any of the following services. vpxa - a service which runs in the esxi host. This service communicates with vcenter server. hostd - core service which runs in the esxi host vpxd - a service which runs in the vcenter server which communicates with the vpxa Resolution: 1. Try to ping the ESXi host. This ensures that the host is reachable 2. Ensure you can login to the VMs remotely. This ensures that VMs in the host are working fine. 3. Try to login to the ESXi host using vsphere client. This ensures that the hostd services are running perfectly. If this step fails, restart the hostd service using putty. 4. If the above three steps worked fine for you, the issue would be with the vpxa or vpxd service. In that case, first try to restart the vpxa service in the host and if that didnot resol
Read more

ESXi : Lost uplink redundancy on virtual switch "vSwitch0". Physical NIC vmnic0 is down

-
We had an issue suddenly getting an alert : Lost uplink redundancy on virtual switch "vSwitch0". Physical NIC vmnic0 is down. Affected portgroups:"vMotion", "Management Network". So as a troubleshooting Select the ESXi host effected Click on Configuration, Select Networking, Click on vSwitch0 Properties,Click on Network adapters Select on vmnic0, Click on remove, Click on Close Again Select Networking, Click on vSwitch0 Properties,Click on Network adapters, Click on Add, Select the vmnic0, Click on Next, Click on Next, Click on Finish Now Click on Ports tab of vSwitch Properties, Select on vMotion, Click on Edit, Click on NIC Teaming, Select on vmnic1, Click on Moveup button  to Active Adapters, Select on vmnic0, Click on Move Down to Standby Adapters, Click on OK Now Click on Ports tab of vSwitch Properties, Select on Management Network, Click on Edit, Click on NIC Teaming, Select on vmnic1, Click on Moveup button  to Active Adapters, Select o
Read more