This article describes use of "Allow Direct FSCopy" Sync option when FREE EXTENTS are NOT available on the Volume Group (VG) , during Linux Server Migration and the important aspects associated with its use.
Background:
Customers occasionally encounter situations during Linux Server Migration, wherein, either the ORIGIN server does not meet Rackware recommended Prereqs by the way of not enough free extents on the Volume Group (VG) available for an LVM Snapshot and/or inability to add recommended free extents, etc.
Symptom:
The RMM Sync fails with an error similar to below
Before you Begin: <none>
Use case:
Migration Sync's of LINUX OS based ORIGIN Servers ONLY, as supported by RW Pre-reqs. This is NOT applicable either to Windows based OS, or Disaster Recovery Sync's.
Applicable To:
ANY RMM version.
Preparation/Pre-Req:
Please refer to section 7.2.2of RMM Pre-Reqs - https://rackware.freshdesk.com/a/solutions/articles/5000859681
Steps/Info:
Rackware's recommended approach prioritizes the presence of an ‘adequate’ amount of free extents on the Volume Group (VG) for all SYNC processes. This ensures that RMM's SYNC process can initiate a 'point in time' LVM snapshot and transfer data from it to maintain data consistency, while ensure Origin server is not impacted.
However, in cases where having sufficient free extents is not possible (or feasible), there is a workaround available for Linux migrations. This workaround, meant to be used for Migration Sync’s (only), involves utilizing the "allow direct fscopy" Sync option, wherein the RMM process will now completely bypass the LVM Snapshot mechanism and directly reads data from the live file system of the Production Server – Hence this option comes with certain conditions and potential risks, as outlined below:
- Data Consistency: Live data reading may lead to inconsistencies in the synchronized data if changes occur during the process, affecting the integrity of the information. Rackware cannot completely assure the data consistency between the ORIGIN and TARGET and cannot be held responsible for any potential issues arising from the use of this option.
- Incomplete or Corrupted Files on Target: Concurrent changes to files while reading live data may result in incomplete or corrupted files during synchronization, leading to potential data loss or errors on the Target.
- Impact on Performance: Live data reading can impact Production system performance, especially in scenarios involving large number of files or large databases, as the synchronization process competes with ongoing operations for resources. This approach can potentially lead to performance degradation on the Production Server and may result in data loss.
- Impact on Production Server (Origin): There is also a rare possibility of data corruption issues occurring on the ORIGIN OS or boot partitions when using this option.
- Impact on Target Server: Can potentially result in the corruption of the OS, boot partitions, applications, databases, or other components on the TARGET server due to inconsistent data.
- Risk of Data Conflicts: Concurrent modifications to files can result in conflicts during synchronization, requiring additional measures to resolve discrepancies and maintain data coherence.
- Extended Synchronization Time: Reading live data without snapshots increases the likelihood of encountering file changes mid-process, causing repeated synchronization attempts and potentially extending the overall synchronization time, especially with large databases.
For instance, if a 1 TB database file undergoes a change, the synchronization process may complete, but the checksum might fail. Consequently, the SYNC process will initiate a restart for that particular file. Should this situation repeat, it could result in an eventual SYNC failure.
It is important to emphasize that this lone event on a 1 TB file has the potential to substantially extend the SYNC duration, consequently prolonging the cutover time.
If it is absolutely imperative to use this option (and no other choice) during Migration sync’s (only – not to be used for Disaster Recovery Sync’s), Rackware recommends certain precautions to be taken:
a) Enable the 'Allow direct fscopy' flag in sync options.
b) It is highly advisable to ‘quiesce’ the application/db and log out other users from the system, to minimize I/O during the SYNC.
c) Preferably, keep the TARGET in our MicroKernel at the end of the SYNC using the "NO REBOOT" Sync option (with a caution to ensure TARGET is NOT rebooted outside of RMM commands).
For executing a 'cutover' or final sync, it is generally recommended to:
a) Preferably ADD enough free extents on each of the VGs on the ORIGIN to allow RMM to take a snapshot and sync the data, and
b) Mandatorily ‘quiesce’ the application/db and log out other users from the system, to minimize I/O during the SYNC.
In essence, following best practices, Rackware emphasizes the critical need for customers to guarantee sufficient free extents in each VG on the ORIGIN servers. The utilization of snapshots during synchronization is imperative to mitigate the mentioned risks, and Rackware strongly discourages bypassing snapshots, especially for pivotal "final" or “cut-over” syncs.
However, it is crucial to note that Rackware does not recommend employing this option during Disaster Recovery Syncs.
(Additional information on this can be found at this link: https://www.tecmint.com/extend-and-reduce-lvms-in-linux/)
This is from our Pre-Req guide - https://rackware.freshdesk.com/a/solutions/articles/5000859681
Post Changes:
Important Note(s):