- Veeam Support Knowledge Base
- Unable to collect files information from datastore in a timely manner due to high vCenter server load
Unable to collect files information from datastore in a timely manner due to high vCenter server load.
| KB ID: | 1653 |
| Product: | Veeam ONE | 6.0 | 6.5 | 7.0 | 8.0 | 9.0 | 9.5 | 10 | 11 | 12 | 12.1 | 12.2 | 12.3 | 13 | 13.1 |
| Published: | 2012-07-06 |
| Last Modified: | 2025-02-04 |
Cheers for trusting us with the spot in your mailbox!
Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest
Oops! Something went wrong.
Please, try again later.
Challenge
Veeam ONE Reporter collects data from datastores using the SearchDatastoreSubFolders_Task tasks, which are pre-defined by vSphere. During that process, if a datastore becomes unavailable or experiences performance issues, Veeam ONE Reporter may be unable to complete data collection tasks against that datastore.
The collection job in Veeam ONE Reporter will finish with a "Warning" status. Additionally, the session log may contain the following entry:
There were errors while collecting VM files: Failed in retrieve some VM files because of the timeout is exceeded.
When this occurs, the following error messages appear in the log file:
C:\ProgramData\Veeam\Veeam ONE Reporter\Logs\Tasks\{hostname}.local.log
Failed to collect VM files: Datastore [{datastore_name}] timeout is exceeded
Unable to collect files information from datastore [{datastore_name}] in a timely manner due to high vCenter server load.
Cause
Veeam ONE Reporter uses a 900-second timeout to wait for 10 datastore search tasks (default number) to complete.
Any performance or access issues affect the entire collection process, and the collection job finishes with a "Warning" status.
The error messages above mean that some issues have occurred on datastores. Due to a lack of response, the datastore scan tasks have been marked as failed, and the data cannot be collected from some of the datastores.
The problem lies either with the datastore or the vSphere environment.
However, there are ways to work around these environmental performance issues by making the collection timeout within Veeam ONE Reporter high enough to allow all datastore tasks to be completed.
Note: This workaround does not fix the underlying environmental issue; it simply changes the SearchDatastoreSubFolders_Task task timeout to ensure adequate time for the datastore collection process to complete.
Solution
To successfully tune the product to complete your datastore search tasks, you must first identify a suitable timeout value for completing all the “SearchDatastoreSubFolders_Task” tasks, then apply registry changes and set the new timeout value.
Part 1: Identify Timeout Value Based on Environment
Using the PowerCLI script below to simulate the “SearchDatastoreSubFolders_Task” tasks and determine how long it takes to complete all mentioned datastore tasks.
Download the VMware vSphere PowerCLI package for free by the following link: https://developer.broadcom.com/powercli
Or install the module using the following command:
Install-Module -Name VMware.PowerCLI
Script Instructions
- Open PowerCLI and connect to the vCenter server using the following command:
Replace<vcenter_name>with the vCenter's hostname, FQDN, or IP.
Connect-VIServer -Server <vcenter_name> -Credential (Get-Credential)
- With the connection to the vCenter server now established, navigate to the directory where you saved the script.
- Run the script and wait until it's finished, noting when it started and when it completes.
Note: This script does not directly reproduce the API calls that Veeam ONE Reporter uses but is a close approximation of the actions it takes.
Expand to View PowerCLI Script
# Display the current time and a message to indicate that datastores are being retrieved
Write-Host (Get-Date).TimeOfDay "Getting datastores..."
# Retrieve the list of all datastores
$datastores = Get-Datastore
# Display the current time and the count of datastores found
Write-Host (Get-Date).TimeOfDay "$($datastores.Count) datastores found."
# Create a FileQueryFlags object to specify which file details to retrieve
$fileQueryFlags = New-Object VMware.Vim.FileQueryFlags
$fileQueryFlags.FileSize = $true
$fileQueryFlags.FileType = $true
$fileQueryFlags.Modification = $true
$fileQueryFlags.FileOwner = $true
# Create a FileQuery object for searching files
$fileQuery1 = New-Object VMware.Vim.FileQuery
# Define a search specification for general file search
$searchSpec1 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$searchSpec1.details = $fileQueryFlags
$searchSpec1.sortFoldersFirst = $true
$searchSpec1.MatchPattern += "*.*" # Match all files
# Define a search specification for finding VMDK-related files
$searchSpec2 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$searchSpec2.details = $fileQueryFlags
$searchSpec2.sortFoldersFirst = $true
$searchSpec2.MatchPattern += "*delta.vmdk" # Match delta VMDK files
$searchSpec2.MatchPattern += "*flat.vmdk" # Match flat VMDK files
# Assign the file query to the general search specification
$searchSpec1.Query = $fileQuery1
# Iterate over each datastore
foreach ($datastore in $datastores) {
# Retrieve the view object of the datastore for detailed access
$datastoreView = $datastore | Get-View
# Check if the datastore is accessible
if ($datastore.Accessible -ne $true) {
# If not accessible, skip file collection and display a message
Write-Host (Get-Date).TimeOfDay "Skipping file collection because $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value)) is not accessible."
} else {
# Retrieve the view object for the datastore browser to perform searches
$datastoreBrowser = Get-View $datastoreView.Browser
# Construct the root path for searching
$rootPath = "[" + $datastoreView.Summary.Name + "]"
# Attempt to collect VMDK-related files
Write-Host (Get-Date).TimeOfDay "Trying to collect *.vmdk and *.flat files from $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
$searchResult2 = $datastoreBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec2)
# Display completion message with the count of files found
Write-Host (Get-Date).TimeOfDay "Completed. $($searchResult2.Count) files found on $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
# Attempt to collect other files
Write-Host (Get-Date).TimeOfDay "Trying to collect other files from $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
$searchResult1 = $datastoreBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec1)
# Display completion message with the count of files found
Write-Host (Get-Date).TimeOfDay "Completed. $($searchResult1.Count) files found on $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
}
}
Part 2: Set the Timeout for Veeam ONE Reporter
Assuming the script took longer than 900 seconds (15 minutes) to complete, adjust the GetVmSnapshotTimeout value to increase the timeout Veeam ONE Reporter uses to complete the entire datastore search process:
- Connect to the machine where Veeam ONE Reporter installed.
- Using Registry Editor (regedit) or the PowerShell script below, set the following registry value:
Key Location: HKLM\SOFTWARE\Veeam\Reporter Enterprise
Value Name: GetVmSnapshotTimeout
Value Type: DWORD (32-Bit) Value
Value Data(Dec): #
Set # to be equal to or greater than time, in seconds, it takes to complete scanning all datastores based on the script's outputted start and end timestamps.
PowerShell:
# Set value in seconds to be used for the GetVmSnapshotTimeout value. Shown is the default.
$time = 900
# Create or update the registry value
New-ItemProperty -Path 'HKLM:\SOFTWARE\Veeam\Reporter Enterprise\' -Name 'GetVmSnapshotTimeout' -Value $time -PropertyType DWORD -Force
If this KB article did not resolve your issue or you need further assistance with Veeam software, please create a Veeam Support Case.
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.