I'm having quite a few problems with operations that act on files in my ESXi 5 environment and my Google foo is not returning anything useful.
* When I try to power on a VM (even one I just created), I get "Failed to create swap file '/sv01/./sv01-79fab788.vswp' : Not found"
I've been able to address this by adding "sched.swap.dir" to the vmx file but why is this occurring especially on a newly created VM?
* When I try to snapshot a VM, I get the following in the VMs logs:
vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)
vcpu-0| I120: Checkpoint_Unstun: vm stopped for 31903 us
vcpu-0| I120: SCSI: switching scsi0 to push completion mode
vcpu-0| I120: Creating virtual dev for scsi0:0
vcpu-0| I120: DumpDiskInfo: scsi0:0 createType=11, capacity = 125829120, numLinks = 1, allocationType = 3
vcpu-0| I120: SCSIDiskESXPopulateVDevDesc: Using FS backend
vcpu-0| I120: DISKUTIL: scsi0:0 : geometry=7832/255/63
vcpu-0| W110: FILE: File_Cwd: getcwd() failed: No such file or directory
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx: Unable to get full path name for 'sv01.vmx'.
vcpu-0| I120: SNAPSHOT: SnapshotConfigInfoReadEx failed for file 'sv01.vmx': Error (3)
vcpu-0| I120: SnapshotVMX_GetDictionaryBuffers: GetDictionaryBuffers failed. Error (3).
vcpu-0| I120: Vix: [10218 mainDispatch.c:6636]: Error VIX_E_FAIL in VMAutomation_AddSnapshotDictBuffersToPropList(): Unknown error
vcpu-0| I120: SnapshotVMXTakeSnapshotCB: Enter checkpoint status 1, mode 2.
vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'test': 0
vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Error (3).
vcpu-0| I120: Vix_TranslateSnapshotError: file IO error = 2
vcpu-0| I120: HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/808e6421-90cf4a4d/sv01/sv01.vmdk'
* When I try to migrate powered off VMs from local storage to a NFS store, I get "Error caused by file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk". In addition, the following errors show up in the vpxa.log:
[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] FileIOErrno2Result: Unexpected errno=9, Bad file descriptor
[30D70B90 info 'Libs' opID=0B209902-000006B4-28-79-7] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/808e6421-90cf4a4d
/sv03/sv03.vmdk' : Bad file descriptor (589827).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (90003)
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LINK : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-CHAIN : "/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk" : failed to open (Bad file descriptor).
[30D70B90 info 'DiskLib' opID=0B209902-000006B4-28-79-7] DISKLIB-LIB : Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk' with flags 0x21f Bad file descriptor (589833).
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFileDskOpenDisk: Failed to open '/vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk': Bad file descriptor (589833).
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_Open: Open failed:
[30D70B90 warning 'Libs' opID=0B209902-000006B4-28-79-7] [NFC ERROR] NfcFile_GetInfo: Failed to open file
[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Unable to get file info for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk: The operation completed successfully
[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] NFC file error for file /vmfs/volumes/808e6421-90cf4a4d/sv03/sv03.vmdk
[30D70B90 verbose 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcClient] Closing NFC connection to server
[30D70B90 error 'NfcManager' opID=0B209902-000006B4-28-79-7] [NfcWorker] Error encountered while pre-processing copy spec for file ds:///vmfs/volumes/47355126-0eb7b838-cbd9-001c23cbbc96/sv03/sv03.vmdk:
--> vim.fault.FileFault
If I log-in to the environment via SSH, I get the same "Bad file descriptor" errors when trying to copy from the shell, however, I was able to use scp from a remote system to copy off the VM without error.
* When I try to host migrate a powered on VM on a NFS store I get the error, "An error occurred while saving the state for migration. Unable to retrieve the current working directory: 0 (No such file or directory). Check if the directory has been deleted or unmounted. Failed to start the virtual machine. Module Migrate power on failed. vMotion migration [-1062731769:1387777483531171] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout"
No amount of restarting the management agents or reboots have helped. To me it appears that the whole environment is corrupt.
Does anyone have any suggestions for where to look for problems or how to correct? The environment is over 1,300 miles away and I have to address all of these issues remotely.