The VM failed to resume on the destination during early power on

Virtual Machine Migration (DRS or user initiated) fails with the following error message:

A genereal system error occurred: The source detected that the destination failed to resume

Error Stack:
The VM failed to resume on the destination during early power on.
Module DiskEarly power on failed.
Cannot open the disk '/vmfs/volumes/<...>.vmdk' or one of the snapshot disks it depends on.
Could not open/create change tracking file

1. Check that Virtual Disks are not mounted to the backup host (VDP or other products). Sometimes when the backup fails, the virtual disks are not correctly removed from the backup host.
2. Consolidate Snapshots (Right Click Virtual Machine > Snapshot > Consolidate)
3. If the problem still persists, create a snapshot and delete it with the "Delete All" option.

I had an interesting issue to resolve today, one that I haven’t seen before and one that took a bit of digging to resolve. The problem related to migrating some Exchange mailbox servers from a legacy ESXi 4.1 host onto new ESXi 5.1 host.

This should have been a simple vMotion operation, but the task failed repeatedly at approximately 65% complete. I tried using both high and standard priority migrations, but it failed every time, simply reporting “The VM failed to resume on the destination during early power on”

First thing I did was check the host log files (vmkwarning and vmkernel), as well as the virtual machine log file (vmware.log) located in the virtual machine folder on the datastore;

So reading through the host log files it looks like there was a problem reserving enough memory resources on the destination host and the operation timed out. This sounds relatively plausible, but the exact same results were observed trying to migrate the VM onto an empty host.

So next step was to review the guest VM’s log file;

Interestingly here, we now start getting some hints that perhaps a file lock is occurring and we also see the same error message that was observed in the vSphere client. The VM failed to resume on the destination during early power on.

I decided to have a look at the contents of the virtual machine folder, and found a number of suspicious looking “-ctk.vmdk” files, mostly time stamped from more than two years ago.

You can consolidate this view with a simple grep of just the “-ctk.vmdk” files;

So for ESX/ESXi 3.x/4.x and ESXi 5.0, the lock status of these “-ctk.vmdk” files can be obtained using the vmkfstools command. The process and syntax is explained in detail in KB1003397, titled “Unable to perform operations on a virtual machine with a locked disk.”

You see output similar to this below, and are specifically interested in the mode value returned;

The mode indicates the type of lock that is on the file, as follows;

In my case, all “-ctk.vmdk” files reported an exclusive mode 1 lock;

So at this stage I created a “tmp” directory in the virtual machine folder and moved all the “-ctk.vmdk” files here. Since this was a live, powered on VM, I felt more comfortable doing this with a GUI than using the shell and used WinSCP to transfer the files.

I then confirmed there were no longer any “-ctk.vmdk” files in the virtual machine folder, and that they were all in the newly created “tmp” folder;

Now the vMotion operation completes successfully;

Here are the accompanying entries from the hosts vmkernel logs, which look much healthier than the original logs;

I’m not sure where these “-ctk.vmdk” files came from, but suspect it may have originated from a legacy backup process from before my time. At least for now the issue is resolved and we know what to look for the next time this happens.

Credits;

Thanks to Jakob Fabritius Nørregaard for posting this blog article which helped identify and resolve this issue.

 81,477 total views,  5 views today