Ensure the following:

VMs should be running the same version of the VMWare tools as the ESX host. ESX should be version 3.5 Update 4 or later (and if a cluster, all hosts should ideally be running the exact same build number. This ensures that when virtual machines are vMotioned to another host that the VMWare tools remain on the proper build for the ESX host.). There are no LUNs attached to the VM via a software iSCSI initiator. Creating a quiesced snapshot of this type of virtual machine is not supported by VMWare per their KB article 1009073 and the corresponding NetApp KB article kb48102 If using SnapDrive for Windows 6.x in a guest, and the OS is Windows Server 2003 R2 or older, a possible workaround is to revert to the SYNC driver. If it is Windows Server 2008 or newer, the SYNC driver is not available, so there is no current workaround other than to not use quiesced VMWare snapshots, or to remove SnapDrive 6.x. See Bug Report 364605

Troubleshooting Steps Take a snapshot outside of SMVI:

This is the first step that should be taken. The instructions for replicating the method in which SMVI calls for a quiesced VMWare Snapshot can be found in NetApp KB article kb55068. a. If this fails, then we know the problem is outside of SMVI. Go to Attempt a nonquiesced snapshot in SMVI b. If this snapshot works, then we may have a filer load issue or communication problem between SMVI and VC. Go to Further Troubleshooting

Attempt a non-quiesced snapshot in SMVI:

In SMVI 1.0.1 or later, there is the option of NOT using VMWare snapshots when taking an SMVI backup. You can reconfigure your jobs or create new jobs that dont include this type of VMWare snapshot like so: 1) For new jobs, during the add job wizard, ensure you UNCHECK the box for Perform VMWare consistency snapshot

2) For existing jobs, edit the job and switch to the Virtual Entities tab (SMVI 1.x) or Entities tab (SMVI 2.x). There, you can clear the checkbox for Perform VMWare consistency snapshot:

Assuming this is successful, this is a possible workaround to the problem. Please note that by using this workaround, although I/O is paused to the VM, the disk buffers are no longer flushed prior to the VMWare snapshot. This results in only crash consistent backups of the VMs. While in a majority of cases this is a recoverable state, it is never guaranteed with this type of backup.

Attempt Snapshot Without VSS Installed:

Similar to the previous step, this process actually removes the VMWare Tools-supplied VSS driver completely. After removing the VSS components, test another snapshot: 1) Open Add/Remove programs in the VM and select VMWare Tools:

2) Click change to bring up the installation wizard. Click Next. Then select modify on the next screen:

3) On the next screen, remove the Filesystem Sync Driver and Volume Shadow Copy Services Support features and click Next (On a Windows Server 2008 box there will not be a Filesystem Sync Driver):

4) Click modify to uninstall these features. Then reboot the VM by shutting it down completely and then restarting it (this creates a new process in ESX rather than continuing the existing one for the given VM). 5) Test an SMVI backup of that VM. As noted in the previous section, the backup will be only crash consistent. 6) Follow steps 1-3 again, this time adding back the Volume Shadow Copy Services Support feature. Reboot the VM and test an SMVI backup once again. If this works, repeat this procedure for all affected VMs. 7) If it does NOT work, your options are: a. Permanently remove the VSS components or uncheck the box in SMVI for Perform VMWare consistency snapshot and simply rely on crash consistent backups. b. If you require file system consistency in your backup, then please open a support case with VMWare technical support. At this point, the issue has been proven to be due to the VSS components not working properly and NOT with SMVI.

Further Troubleshooting:
If the above scenarios do not help isolate the problem, there may be either an underlying storage performance issue, or some other problem with SMVI. The VMWare Snapshot process can fail due to storage performance problems. In order to confirm/deny that performance is an issue, collect a perfstat during the SMVI backup job. Information about this process can be found in Performance and Statistics Collector (perfstat). In addition, NetApp Global Support is available to assist with proper syntax and methodology for performance data collection. The general recommendation is to use several iterations of 5 minutes in length while the problematic SMVI job is occurring. This will allow NetApp Global Support to analyze filer performance at a very detailed level specifically while the SMVI job was running. Again, please contact NetApp Global Support for assistance with perfstat collection if you need assistance. If the above troubleshooting processes do not lead to resolution, then further analysis is required by NetApp Global Support. Please work with NGS to collect the required data to further troubleshoot the issue. Data collection for SMVI troubleshooting should include the following: 1. Begin by collecting verbose SMVI logs per this NetApp KB Article kb45492 2. Generate a VMWare Support Bundle using the vSphere or VI Client:

Virtual Center 2.5/VI3: 1. Open the VI Client and connect to vCenter. 2. Go to the Administration Menu and select Export Diagnostic Data.

3. Check the box to highlight all ESX servers in the cluster and make sure the box entitled Include information from VirtualCenter Server and VI Client is checked:

vCenter 4/ESX 4.0: 1. Open the vSphere client and connect to vCenter . 2. Go to Administration > Export System Logs. 3. Check the box to highlight all ESX servers in the cluster and make sure the box entitled Include information from vCenter Server and vSphere Client is checked: