The pvmove command fails on clustered LVM volume groups

This document (7022924) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise High Availability Extension 12 (HAE 12)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

Running pvmove on an active clustered LVM device shows the move complete on some nodes, but stuck on another node; usually the initiating node.
node1:~ # lvs -a -o +devices
 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices lv_root vg00 -wi-ao---- 27.86g /dev/sda1(0) lv_swap vg00 -wi-ao---- 4.00g /dev/sda1(70108) lv_images vg_images -wI-ao---- 4.00t pvmove0(0) [pvmove0] vg_images p-C-aom--- 4.00t /dev/mapper/IMAGES1 3.36 /dev/mapper/IMAGES1(0),/dev/mapper/IMAGES2(0) node2:~ # lvs -a -o +devices
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices lv_root vg00 -wi-ao---- 27.86g /dev/sda1(0) vg_swap vg00 -wi-ao---- 4.00g /dev/sda1(70108) lv_images vg_images -wI-ao---- 4.00t pvmove0(0) [pvmove0] vg_images p-C-aom--- 4.00t /dev/mapper/IMAGES1 100.00 /dev/mapper/IMAGES1(0),/dev/mapper/IMAGES2(0)
The volume group was experiencing heavy I/O load.

Resolution

There are three possible solutions:

1. Put all but one node in stand-by mode, perform the pvmove command, then activate the nodes again
Advantages: The ocfs2 filesystem remains mounted with resources still running
Disadvantages: All resources are running on one node during the pvmove operation
Procedure:
a. On a three node cluster run:
  crm_standby --node=node3 --update=on
  crm_standby --node=node2 --update=on
b. From node1, run pvmove /dev/mapper/IMAGES1
c. vgreduce vg-images /dev/mapper/IMAGES1
d. pvremove /dev/mapper/IMAGES1
e. Activate the standby nodes:
  crm_standby --node=node3 --update=off
  crm_standby --node=node2 --update=off

2. Wait until there is little I/O to the volume and attempt the pvmove.
Advantages: The ocfs2 filesystem remains mounted with resources running across all nodes.
Disadvantages: The pvmove command is not fully cluster aware, so I/O load would have to be minimal. There is still a risk the pvmove command will hang.
Procedure:
a. Wait for I/O to be minimal.
b. pvmove /dev/mapper/IMAGES1
c. vgreduce vg-images /dev/mapper/IMAGES1
d. pvremove /dev/mapper/IMAGES1

3. Stop the filesystem resource, run pvmove on one node and restart the filesystem resource.
Advantages: The LVM volume is isolated to one node and the pvmove command has all it's requirements met.
Disadvantages: Requires clone filesystem downtime during the pvmove.
Procedure:
a. Stop the clustered filesystem resource. This will cause the filesystem to be unmounted and the resources using that filesystem should stop.
b. pvmove /dev/mapper/IMAGES1
c. vgreduce vg-images /dev/mapper/IMAGES1
d. pvremove /dev/mapper/IMAGES1
e. Start the clustered filesystem resource.

Cause

The pvmove command is not completely cluster aware. It should be run on an isolated node.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7022924
  • Creation Date: 04-May-2018
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise High Availability Extension
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center