Issue:
Using "rwadm zfs configure" to add a disk to the zfs pool is unsuccessful. The response returned says that the disk device is in use and contains an unknown file system even though it was earlier reported as being free, and the space in the disk is not added to the zfs pool.
Before you Begin:
Ensure you have ROOT access to RMM via SSH. Use the 'parted -l' command to verify that the disk you wish to add to the zfs pool is visible to the OS. In this case /dev/sda is the 60 GB disk that is to be used for the zfs pool, and /dev/sdb is the boot volume.
[root@bob-from-marketplace-74 opc]# parted -l
Error: /dev/sda: unrecognised disk label
Model: ORACLE BlockVolume (scsi)
Disk /dev/sda: 60GB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags:
Model: ORACLE BlockVolume (scsi)
Disk /dev/sdb: 116GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 106MB 105MB fat16 EFI System Partition boot, esp
2 106MB 1180MB 1074MB xfs
3 1180MB 107GB 106GB
4 107GB 116GB 8590MB
So /dev/sda is visible to the OS
Initial Attempt:
The 'rwadm zfs configure' command is used to add a disk to the zfs pool. The RMM in this example came from the OCI Marketplace, and already had 7.5 GB of space in the zfs pool. But we would like to add the 60 GB disk (/dev/sda) to the pool. Below is the output of the initial 'rwadm zfs configure' command:
[root@bob-from-marketplace-74 opc]# rwadm zfs configure
Red Hat Enterprise Linux release 8.9 (Ootpa)
Found supported RedHat/CentOS release.
Red Hat Enterprise Linux release 8.9 (Ootpa)
===============================================================
Available storages
1. default at localhost:/srv/images
Enter your choice [1]:
RMM Storage Pool is already configured
Statistics of RMM Storage pool is :
=====================================================================
| RMM STORAGE POOL |
=====================================================================
Pool Name : rwzpool
Total Size : 7.50G
Pool Free : 7.50G
ZFS Compression Algorithm: lz4
=====================================================================
Do you want to add extra disk to RMM storage pool? (Y/N) [N]: y
Existing devices in the system which can be added to RMM Storage Pool are:
=====================================================================
| EXISTING DEVICES |
=====================================================================
/dev/sdb (in-use)
/dev/sdb1 (in-use)
/dev/sdb2 (in-use)
/dev/sdb4 (in-use for ZFS)
/dev/sdb3 (in-use for LVM)
/dev/sda (free)
=====================================================================
[A]dd disk or [F]inished [A]: A
You have 3 chance(s) to enter a valid device. Check by executing "parted -l"
Warning: Device will get formatted after adding to RMM Storage pool.
Enter device name/path to be configured as RMM Storage pool. [ONE AT A TIME]: /dev/sda
Valid device: /dev/sda
Following disks will be added to RMM Storage Pool:
/dev/sda
/dev/sda is in use and contains a unknown filesystem.
Status of rwzpool:
pool: rwzpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
rwzpool ONLINE 0 0 0
sdb4 ONLINE 0 0 0
errors: No known data errors
Existing devices in the system which can be added to RMM Storage Pool are:
=====================================================================
| EXISTING DEVICES |
=====================================================================
/dev/sdb (in-use)
/dev/sdb1 (in-use)
/dev/sdb2 (in-use)
/dev/sdb4 (in-use for ZFS)
/dev/sdb3 (in-use for LVM)
/dev/sda (free)
=====================================================================
[A]dd disk or [F]inished [A]: F
Retaining Previous Configuration
Loading ZFS properties from previous configuration...
Enabling ZFS Compression...
Final Configuration:
=====================================================================
| RMM STORAGE POOL |
=====================================================================
Pool Name : "rwzpool"
Total Size : 7.50G
Pool Free : 7.50G
RMM Storage Pool Compression Algorithm: "lz4"
Note that even though /dev/sda was initially reported as 'free', when it was entered as the zfs disk the response was '/dev/sda is in use and contains a unknown filesystem'. Thus it was not added to the zfs pool and pool size remained the same as it was initially.
Debugging Steps:
Issue the lsblk command
[root@bob-from-marketplace-74 opc]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 60G 0 disk
└─3608d49184400456bbbb57785a3be6d25 252:2 0 60G 0 mpath
sdb 8:16 0 108G 0 disk
├─sdb1 8:17 0 100M 0 part /boot/efi
├─sdb2 8:18 0 1G 0 part /boot
├─sdb3 8:19 0 98.9G 0 part
│ ├─ocivolume-root 252:0 0 88.9G 0 lvm /
│ └─ocivolume-oled 52:1 0 10G 0 lvm /var/oled
└─sdb4 8:20 0 8G 0 part
Note that the device type under sda is mpath.
The fact that this is a multipath device is why /dev/sda was considered "in use"
The "multipath -ll" command also shows that the disk device is a multipath device
[root@bob-mktplace-74-4 opc]# multipath -ll
3608d49184400456bbbb57785a3be6d25 dm-2 ORACLE,BlockVolume
size=66G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
`- 0:0:1:1 sda 8:0 active ready running
Solution:
In general, there should not be multipath devices on the RMM. So the easiest way to fix this is to use the "multipath -F" command to remove all multipath devices on the RMM
[root@bob-mktplace-74-4 opc]# multipath -F
[root@bob-mktplace-74-4 opc]#
Now the lsblk command will show /dev/sda as a regular disk device
[root@bob-from-marketplace-74 opc]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 60G 0 disk
sdb 8:16 0 108G 0 disk
├─sdb1 8:17 0 100M 0 part /boot/efi
├─sdb2 8:18 0 1G 0 part /boot
├─sdb3 8:19 0 98.9G 0 part
│ ├─ocivolume-root 252:0 0 88.9G 0 lvm /
│ └─ocivolume-oled 52:1 0 10G 0 lvm /var/oled
└─sdb4 8:20 0 8G 0 part
Now when 'rwadm zfs configure' is run, /dev/sda will be added to the zfs pool
[root@bob-from-marketplace-74 opc]# rwadm zfs configure
Red Hat Enterprise Linux release 8.9 (Ootpa)
Found supported RedHat/CentOS release.
Red Hat Enterprise Linux release 8.9 (Ootpa)
===============================================================
Available storages
1. default at localhost:/srv/images
Enter your choice [1]:
RMM Storage Pool is already configured
Statistics of RMM Storage pool is :
=====================================================================
| RMM STORAGE POOL |
=====================================================================
Pool Name : rwzpool
Total Size : 7.50G
Pool Free : 7.50G
ZFS Compression Algorithm: lz4
=====================================================================
Do you want to add extra disk to RMM storage pool? (Y/N) [N]: y
Existing devices in the system which can be added to RMM Storage Pool are:
=====================================================================
| EXISTING DEVICES |
=====================================================================
/dev/sdb (in-use)
/dev/sdb1 (in-use)
/dev/sdb2 (in-use)
/dev/sdb4 (in-use for ZFS)
/dev/sdb3 (in-use for LVM)
/dev/sda (free)
=====================================================================
[A]dd disk or [F]inished [A]: a
You have 3 chance(s) to enter a valid device. Check by executing "parted -l"
Warning: Device will get formatted after adding to RMM Storage pool.
Enter device name/path to be configured as RMM Storage pool. [ONE AT A TIME]: /dev/sda
Valid device: /dev/sda
Following disks will be added to RMM Storage Pool:
/dev/sda
Status of rwzpool:
pool: rwzpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
rwzpool ONLINE 0 0 0
sdb4 ONLINE 0 0 0
sda ONLINE 0 0 0
errors: No known data errors
Existing devices in the system which can be added to RMM Storage Pool are:
=====================================================================
| EXISTING DEVICES |
=====================================================================
/dev/sdb (in-use)
/dev/sdb1 (in-use)
/dev/sdb2 (in-use)
/dev/sdb4 (in-use for ZFS)
/dev/sda (in-use for ZFS)
/dev/sda1 (in-use for ZFS)
/dev/sdb3 (in-use for LVM)
=====================================================================
[A]dd disk or [F]inished [A]: f
Retaining Previous Configuration
Loading ZFS properties from previous configuration...
Enabling ZFS Compression...
Final Configuration:
=====================================================================
| RMM STORAGE POOL |
=====================================================================
Pool Name : "rwzpool"
Total Size : 67G
Pool Free : 67.0G
RMM Storage Pool Compression Algorithm: "lz4"
So the space in /dev/sda has been added to the zfs pool
Alternate Solutions:
If for some reason "multipath -F" can not be used in your environment, another option would be to just remove multipathing on the one disk with "multipath -f /dev/mapper/<uuid>", which in the example above would be
"multipath -f /dev/mapper/3608d49184400456bbbb57785a3be6d25"
Rather than changing the multipath settings, it is also possible to add the disk to the zfs pool by specifying it as "/dev/mapper/<uuid>" when running 'rwadm zfs configure'. So, in the example above, when running 'rwadm zfs configure', instead of specifying "/dev/sda" as the disk to add, specify "/dev/mapper/3608d49184400456bbbb57785a3be6d25" as the disk to add.
Removing multipathing is, however, the recommended solution.