While there are several factors that can cause slow performance on Windows syncs, the most common reasons for slow throughput are the configuration of the AntiVIrus software on the WIndows server and the I/O capacity of the Windows server when reading from a snapshot.     This article will discuss both of these factors and provide a way to measure the I/O capacity of the Windows server when reading data from a snapshot.


1) AntiVirus Configuration

If RMM processes  are not whitelisted from the antivirus scan, it can further slow down the sync process. Please make sure the exclusions are in place. These RMM processes will perform a large number of random reads of files and directories on the file system being synced. Most on-access antivirus products intercept I/O activity performed by processes running on a machine. These interceptions do not always result in the process being blocked from performing its actions. But they usually do introduce some latency into the I/O path. Because of the large number of I/O operations these RMM processes must perform, even a small amount of added latency to each I/O operation can add up to a large amount of total time spent waiting for antivirus to complete its intercepts and can result in drastically reduced sync performance.

To avoid the performance problems this can cause, the antivirus product should be configured to not intercept the I/O operations generated by these RMM processes. Sometimes these are referred to as "process exclusions" (i.e. any I/O generated by that process will be excluded from antivirus on-demand scanning). But some products may use different terminology.


Please see section 7.3.3 of the  RMM Prerequisites and Operational Requirements manual for the antivirus whitelisting that must be in place.


Not having the Antivirus whitelisting in place will affect all phases of the sync.   Some phases have more variables involved than others that determine the performance being obtained, but the most measurable part of a sync (if not using the TNG feature) is the 'Syncing NTFS metadata' (or rwattr) phase.   This is the phase of the sync where the RMM transfers the attributes of the files and directories being synced.     The by-jobid log shows when this phase of the sync began, and when it ends, and how many files and directories had their attributes transferred.  


Even on the Windows servers with very slow IO, if the AntiVirus whitelisting is in place, there will be at least 3,000,000 file and directory attributes transferred per hour.    If you see that there are less than 10,000,000 file and directory attributes transferred per hour it is worthwhile to recheck the Antivirus settings.  If you see that there are less than 3,000,000 file and directory attributes transferred per hour, it is almost certainly the case that the Antivirus whitelisting is NOT in place.

To determine these values from the log, search on "TRANSFER_RWATTR".    Note the time of this record.  Then search on "TRANSFER_RWATTR_RESULT".   Then scroll up around 20 lines and you will see a record like

" Total Pass: 548113   Pass Files: 461063   Pass Dirs: 87050"    Note the time of this record, and the value of 'Total Pass'.    Then, using the time of the TRANSFER_RWATTR record, and the time of the "Total Pass" record, and the value of 'Total Pass', calculate the transfer rate.     In the example above, if there was 30 minutes between TRANSFER_RWATT and "Total Pass", then the transfer rate is 548,113 * 2 = 1,096,226 files and directories per hour, so it would be very likely that the Antivirus whitelisting is not set correctly.


If there are multiple volumes on the Windows host, there will be a set of records like the above for each volume.   Perform the calculation on the volume that has the most files and directories.


2) I/O Capacity of Windows Server When Reading from a Snapshot

The network throughput or the I/O capacity of the Windows host might also be a performance bottleneck.  The attached file discusses a method for determining the network throughput of an ssh session and a method for determining the I/O capacity of the Windows server when reading data from a Snapshot, both of which the RMM does while syncing data.  These values indicate the maximum theoretical throughput of the RMM.     Like any application, the actual RMM throughput will always be less than the maximum theoretical throughput.


Related Reference(s):

https://rackware.freshdesk.com/a/solutions/articles/5000885786