This article describes the steps needed to move the directory /var/log/rackware/ from the / partition to the rwzpool/logs in the zfs pool.
Background: On an RMM server, often the /var directory is part of the / partition, and the / partition is fairly small, for example, 40 GB. If the RMM is being used for Disaster Recovery (DR), and syncs are happening frequently, there will be many files each day in the /var/log/rackware/by-jobid/ directory, which may rapidly use up the space in the (small) / partition. If / ever runs out of space the RMM will fail to function properly. So running out of space in / should be avoided.
One way of doing this would be to reduce the number of days that log files are kept on the RMM through modifications to the /opt/rackware/scripts/logrotate script. But doing this might cause a log file to be removed that later on is determined to be a log file with important information in it.
If the RMM is being used for DR, then you already have one or more terabytes of space in a ZFS pool, While the size needed for /var/log/rackware may be large relative to the size of the / partition, it will be small relative to the size of the zfs pool.
If the RMM is being used for Migrations, and the / partition is unusually small so you wish to move it to a zfs pool, you can create a zfs pool with the "rwadm zfs configure" command as described in Chapter 5 of https://rackware.freshdesk.com/a/solutions/articles/5000870805. Note that you will first need to have a free disk available for the zfs storage.
Before Beginning: Be sure you have root access to the RMM. It is possible that a few records in a log file that is being written to could be lost while moving the log files.
Procedure:
1) Issue the commands "df -h /" and "zfs list". Save the output of these commands. The output of these commands now will be compared with the output of those commands and the end of this procedure.
Then issue the following commands:
2) cd /
3)zfs create rwzpool/logs
4)zfs set compression=on rwzpool/logs
5)systemctl stop syslog.socket rsyslog.service
6)mv /var/log/rackware/* /rwzpool/logs (this step might take a few minutes)
7)zfs set mountpoint=/var/log/rackware rwzpool/logs
8)systemctl start syslog.socket rsyslog.service
9)rwadm restart httpd
10) Now reissue the commands "df -h /" and "zfs list". The output of the df command
will show that there is now more free space in /. The output of the "zfs list"
command will now show a "rwzpool/logs" entry
[root@rackware-rmm by-jobid]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rwzpool 6.68T 3.69T 96K /rwzpool
rwzpool/d067dd55-d33c-11ed-95d7-00001702bae4 875G 3.69T 691G /srv/images/d067dd55-d33c-11ed-95d7-00001702bae4
rwzpool/d06a1d36-d33c-11ed-95d7-00001702bae4 5.81T 3.69T 583G /srv/images/d06a1d36-d33c-11ed-95d7-00001702bae4
rwzpool/logs 5.02G 3.69T 5.02G /var/log/rackware
At this point the logs have been moved, but some cleanup steps still need to be done.
Cleanup Steps:
1) lsof | grep log/rackware
This will show a list of processes and files such as:
[root@rackware-rmm by-jobid]# lsof | grep log/rackware
rsyslogd 672 root 9w REG 8,51 9295043 67599662 /var/log/rackware/rmm.log (deleted)
rsyslogd 672 root 10w REG 8,51 1024385 41677301 /var/log/rackware/by-jobid/0000000000011363.log (deleted)
rsyslogd 672 root 12w REG 8,51 6647397 69657901 /var/log/rackware/autoscale.log (deleted)
rsyslogd 672 root 13w REG 8,51 6647397 69657901 /var/log/rackware/autoscale.log (deleted)
rsyslogd 672 root 14w REG 0,41 9930 564 /var/log/rackware/by-jobid/000000000001136e.log
rsyslogd 672 root 16w REG 8,51 1733689 69035497 /var/log/rackware/auth.log (deleted)
rsyslogd 672 root 17w REG 8,51 3856434 41677300 /var/log/rackware/by-jobid/0000000000011362.log (deleted)
rsyslogd 672 root 19w REG 8,51 3559831 70418436 /var/log/rackware/rw.log (deleted)
rsyslogd 672 root 20w REG 8,51 6067517 67599666 /var/log/rackware/multitenancy.log (deleted)
rsyslogd 672 root 21w REG 0,41 9249 565 /var/log/rackware/by-jobid/0000000000011370.log
rsyslogd 672 root 22w REG 8,51 1022209 41621373 /var/log/rackware/by-jobid/0000000000011361.log (deleted)
rsyslogd 672 root 23w REG 8,51 1733689 69035497 /var/log/rackware/auth.log (deleted)
rsyslogd 672 root 24w REG 8,51 1376703 44113964 /var/log/rackware/by-jobid/0000000000010b92.log (deleted)
rsyslogd 672 root 25w REG 0,41 9249 561 /var/log/rackware/by-jobid/000000000001136d.log
rsyslogd 672 root 26w REG 8,51 1008446 43623292 /var/log/rackware/by-jobid/00000000000111d8.log (deleted)
rsyslogd 672 root 27w REG 8,51 4578909 44114539 /var/log/rackware/by-jobid/0000000000000000.log (deleted)
rsyslogd 672 root 28w REG 8,51 1389005 40805933 /var/log/rackware/by-jobid/0000000000011360.log (deleted)
rsyslogd 672 root 29w REG 8,51 6647397 69657901 /var/log/rackware/autoscale.log (deleted)
in:imuxso 672 673 root 9w REG 8,51 9295043 67599662 /var/log/rackware/rmm.log (deleted)
in:imuxso 672 673 root 10w REG 8,51 1024385 41677301 /var/log/rackware/by-jobid/0000000000011363.log (deleted)
in:imuxso 672 673 root 12w REG 8,51 6647397 69657901 /var/log/rackware/autoscale.log (deleted)
in:imuxso 672 673 root 13w REG 8,51 6647397 69657901 /var/log/rackware/autoscale.log (deleted)
in:imuxso 672 673 root 14w REG 0,41 9930 564 /var/log/rackware/by-jobid/000000000001136e.log
in:imuxso 672 673 root 16w REG 8,51 1733689 69035497 /var/log/rackware/auth.log (deleted)
in:imuxso 672 673 root 17w REG 8,51 3856434 41677300 /var/log/rackware/by-jobid/0000000000011362.log (deleted)
in:imuxso 672 673 root 19w REG 8,51 3559831 70418436 /var/log/rackware/rw.log (deleted)
in:imuxso 672 673 root 20w REG 8,51 6067517 67599666 /var/log/rackware/multitenancy.log (deleted)
in:imuxso 672 673 root 21w REG 0,41 9249 565 /var/log/rackware/by-jobid/0000000000011370.log
in:imuxso 672 673 root 22w REG 8,51 1022209 41621373 /var/log/rackware/by-jobid/0000000000011361.log (deleted)
in:imuxso 672 673 root 23w REG 8,51 1733689 69035497 /var/log/rackware/auth.log (deleted)
in:imuxso 672 673 root 24w REG 8,51 1376703 44113964 /var/log/rackware/by-jobid/0000000000010b92.log (deleted)
Note that there are several Names that have "(deleted)" after them. We want to kill the process that owns all those files, which is 672 in this case.
2) kill -1 672
[root@rackware-rmm /]# kill -1 672
[root@rackware-rmm /]#
Please note that the switch in the kill command is the number "1", not the letter "l".
Now we will restart the syslog process
3) rwadm restart syslog
[root@rackware-rmm /]# rwadm restart syslog
CentOS Linux release 7.8.2003 (Core)
Found supported RedHat/CentOS release.
CentOS Linux release 7.8.2003 (Core)
Stopping: ... Done.
* running: rsyslogd[672]
Starting: ... Done.
* running: rsyslogd[672]
This completes the procedure for moving the space needed for /var/log/rackware/ to the zfs pool.