7.3 Configuring Cluster Resources

As a cluster administrator, you need to create cluster resources for every resource or application you run on servers in your cluster. Cluster resources can include Web sites, e-mail servers, databases, file systems, virtual machines, and any other server-based applications or services you want to make available to users at all times.

For an overview of resource types you can create, refer to Section 4.2.3, Types of Resources.

7.3.1 Creating Cluster Resources

There are three types of RAs (Resource Agents) available with the cluster (for background information, see Section 4.2.2, Supported Resource Agent Classes). To add a new resource to the cluster, proceed as follows:

  1. Log in as root and start the crm tool:

    root # crm configure
  2. Configure a primitive IP address:

    crm(live)configure# primitive myIP ocf:heartbeat:IPaddr \
         params ip=127.0.0.99 op monitor interval=60s

    The previous command configures a primitive with the name myIP. You need to choose a class (here ocf), provider (heartbeat), and type (IPaddr). Furthermore, this primitive expects other parameters like the IP address. Change the address to your setup.

  3. Display and review the changes you have made:

    crm(live)configure# show
  4. Commit your changes to take effect:

    crm(live)configure# commit

7.3.2 Creating Resource Templates

If you want to create several resources with similar configurations, a resource template simplifies the task. See also Section 4.4.3, Resource Templates and Constraints for some basic background information. Do not confuse them with the normal templates from Section 7.1.4, Using Configuration Templates. Use the rsc_template command to get familiar with the syntax:

root # crm configure rsc_template
usage: rsc_template <name> [<class>:[<provider>:]]<type>
        [params <param>=<value> [<param>=<value>...]]
        [meta <attribute>=<value> [<attribute>=<value>...]]
        [utilization <attribute>=<value> [<attribute>=<value>...]]
        [operations id_spec
            [op op_type [<attribute>=<value>...] ...]]

For example, the following command creates a new resource template with the name BigVM derived from the ocf:heartbeat:Xen resource and some default values and operations:

crm(live)configure# rsc_template BigVM ocf:heartbeat:Xen \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

Once you defined the new resource template, you can use it in primitives or reference it in order, colocation, or rsc_ticket constraints. To reference the resource template, use the @ sign:

crm(live)configure# primitive MyVM1 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"

The new primitive MyVM1 is going to inherit everything from the BigVM resource templates. For example, the equivalent of the above two would be:

crm(live)configure# primitive MyVM1 ocf:heartbeat:Xen \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1" \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

If you want to overwrite some options or operations, add them to your (primitive) definition. For instance, the following new primitive MyVM2 doubles the timeout for monitor operations but leaves others untouched:

crm(live)configure# primitive MyVM2 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM2" name="MyVM2" \
   op monitor timeout=120s interval=30s    

A resource template may be referenced in constraints to stand for all primitives which are derived from that template. This helps to produce a more concise and clear cluster configuration. Resource template references are allowed in all constraints except location constraints. Colocation constraints may not contain more than one template reference.

7.3.3 Creating a STONITH Resource

From the crm perspective, a STONITH device is just another resource. To create a STONITH resource, proceed as follows:

  1. Log in as root and start the crm interactive shell:

    root # crm configure
  2. Get a list of all STONITH types with the following command:

    crm(live)# ra list stonith
    apcmaster                  apcmastersnmp              apcsmart
    baytech                    bladehpi                   cyclades
    drac3                      external/drac5             external/dracmc-telnet
    external/hetzner           external/hmchttp           external/ibmrsa
    external/ibmrsa-telnet     external/ipmi              external/ippower9258
    external/kdumpcheck        external/libvirt           external/nut
    external/rackpdu           external/riloe             external/sbd
    external/vcenter           external/vmware            external/xen0
    external/xen0-ha           fence_legacy               ibmhmc
    ipmilan                    meatware                   nw_rpc100s
    rcd_serial                 rps10                      suicide
    wti_mpc                    wti_nps
  3. Choose a STONITH type from the above list and view the list of possible options. Use the following command:

    crm(live)# ra info stonith:external/ipmi
    IPMI STONITH external device (stonith:external/ipmi)
    
    ipmitool based power management. Apparently, the power off
    method of ipmitool is intercepted by ACPI which then makes
    a regular shutdown. If case of a split brain on a two-node
    it may happen that no node survives. For two-node clusters
    use only the reset method.
    
    Parameters (* denotes required, [] the default):
    
    hostname (string): Hostname
        The name of the host to be managed by this STONITH device.
    ...
  4. Create the STONITH resource with the stonith class, the type you have chosen in , and the respective parameters if needed, for example:

    crm(live)# configure
    crm(live)configure# primitive my-stonith stonith:external/ipmi \
        params hostname="alice" \
        ipaddr="192.168.1.221" \
        userid="admin" passwd="secret" \
        op monitor interval=60m timeout=120s  

7.3.4 Configuring Resource Constraints

Having all the resources configured is only one part of the job. Even if the cluster knows all needed resources, it might still not be able to handle them correctly. For example, try not to mount the file system on the slave node of DRBD (in fact, this would fail with DRBD). Define constraints to make these kind of information available to the cluster.

For more information about constraints, see Section 4.4, Resource Constraints.

Locational Constraints

The location command defines on which nodes a resource may be run, may not be run or is preferred to be run.

This type of constraint may be added multiple times for each resource. All location constraints are evaluated for a given resource. A simple example that expresses a preference to run the resource fs1 on the node with the name alice to 100 would be the following:

crm(live)configure# location loc-fs1 fs1 100: alice

Another example is a location with pingd:

crm(live)configure# primitive pingd pingd \
    params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
crm(live)configure# location loc-node_pref internal_www \
    rule 50: #uname eq alice \
    rule pingd: defined pingd

Another use case for location constraints are grouping primitives as a resource set. This can be useful if several resources depend on, for example, a ping attribute for network connectivity. In former times, the -inf/ping rules needed to be duplicated several times in the configuration, making it unnecessarily complex.

The following example creates a resource set loc-alice, referencing the virtual IP addresses vip1 and vip2:

crm(live)configure# primitive vip1 ocf:heartbeat:IPaddr2 params ip=192.168.1.5
crm(live)configure# primitive vip1 ocf:heartbeat:IPaddr2 params ip=192.168.1.6
crm(live)configure# location loc-alice { vip1 vip2 } inf: alice 

In some cases it is much more efficient and convenient to use resource patterns for your location command. A resource pattern is a regular expression between two slashes. For example, the above virtual IP addresses can be all matched with the following:

crm(live)configure# location  loc-alice /vip.*/ inf: alice

Colocational Constraints

The colocation command is used to define what resources should run on the same or on different hosts.

It is only possible to set a score of either +inf or -inf, defining resources that must always or must never run on the same node. It is also possible to use non-infinite scores. In that case the colocation is called advisory and the cluster may decide not to follow them in favor of not stopping other resources if there is a conflict.

For example, to run the resources with the IDs filesystem_resource and nfs_group always on the same host, use the following constraint:

crm(live)configure# colocation nfs_on_filesystem inf: nfs_group filesystem_resource

For a master slave configuration, it is necessary to know if the current node is a master in addition to running the resource locally.

Collocating Sets for Resources Without Dependency

Sometimes it is useful to be able to place a group of resources on the same node (defining a colocation constraint), but without having hard dependencies between the resources.

Use the command weak-bond if you want to place resources on the same node, but without any action if one of them fails.

root # crm configure assist weak-bond RES1 RES2

The implementation of weak-bond creates a dummy resource and a colocation constraint with the given resources automatically.

Ordering Constraints

The order command defines a sequence of action.

Sometimes it is necessary to provide an order of resource actions or operations. For example, you cannot mount a file system before the device is available to a system. Ordering constraints can be used to start or stop a service right before or after a different resource meets a special condition, such as being started, stopped, or promoted to master.

Use the following command in the crm shell to configure an ordering constraint:

crm(live)configure# order nfs_after_filesystem mandatory: filesystem_resource nfs_group

Constraints for the Example Configuration

The example used for this section would not work without additional constraints. It is essential that all resources run on the same machine as the master of the DRBD resource. The DRBD resource must be master before any other resource starts. Trying to mount the DRBD device when it is not the master simply fails. The following constraints must be fulfilled:

  • The file system must always be on the same node as the master of the DRBD resource.

    crm(live)configure# colocation filesystem_on_master inf: \
        filesystem_resource drbd_resource:Master
  • The NFS server and the IP address must be on the same node as the file system.

    crm(live)configure# colocation nfs_with_fs inf: \
       nfs_group filesystem_resource
  • The NFS server and the IP address start after the file system is mounted:

    crm(live)configure# order nfs_second mandatory: \
       filesystem_resource:start nfs_group
  • The file system must be mounted on a node after the DRBD resource is promoted to master on this node.

    crm(live)configure# order drbd_first inf: \
        drbd_resource:promote filesystem_resource:start

7.3.5 Specifying Resource Failover Nodes

To determine a resource failover, use the meta attribute migration-threshold. In case failcount exceeds migration-threshold on all nodes, the resource will remain stopped. For example:

crm(live)configure# location rsc1-alice rsc1 100: alice

Normally, rsc1 prefers to run on alice. If it fails there, migration-threshold is checked and compared to the failcount. If failcount >= migration-threshold then it is migrated to the node with the next best preference.

Start failures set the failcount to inf depend on the start-failure-is-fatal option. Stop failures cause fencing. If there is no STONITH defined, the resource will not migrate.

For an overview, refer to Section 4.4.4, Failover Nodes.

7.3.6 Specifying Resource Failback Nodes (Resource Stickiness)

A resource might fail back to its original node when that node is back online and in the cluster. If you want to prevent a resource from failing back to the node that it was running on, or if you want to specify a different node for the resource to fail back to, change its resource stickiness value. You can either specify resource stickiness when you are creating a resource, or afterwards.

For an overview, refer to Section 4.4.5, Failback Nodes.

7.3.7 Configuring Placement of Resources Based on Load Impact

Some resources may have specific capacity requirements such as minimum amount of memory. Otherwise, they may fail to start completely or run with degraded performance.

To take this into account, the High Availability Extension allows you to specify the following parameters:

  1. The capacity a certain node provides.

  2. The capacity a certain resource requires.

  3. An overall strategy for placement of resources.

For detailed background information about the parameters and a configuration example, refer to Section 4.4.6, Placing Resources Based on Their Load Impact.

To configure the resource's requirements and the capacity a node provides, use utilization attributes. You can name the utilization attributes according to your preferences and define as many name/value pairs as your configuration needs. In certain cases, some agents update the utilization themselves, for example the VirtualDomain.

In the following example, we assume that you already have a basic configuration of cluster nodes and resources. You now additionally want to configure the capacities a certain node provides and the capacity a certain resource requires.

Adding Or Modifying Utilization Attributes With crm

  1. Log in as root and start the crm interactive shell:

    root # crm configure
  2. To specify the capacity a node provides, use the following command and replace the placeholder NODE_1 with the name of your node:

    crm(live)configure# node
    NODE_1 utilization memory=16384 cpu=8

    With these values, NODE_1 would be assumed to provide 16GB of memory and 8 CPU cores to resources.

  3. To specify the capacity a resource requires, use:

    crm(live)configure# primitive xen1 ocf:heartbeat:Xen ... \
         utilization memory=4096 cpu=4

    This would make the resource consume 4096 of those memory units from nodeA, and 4 of the CPU units.

  4. Configure the placement strategy with the property command:

    crm(live)configure# property ...

    The following values are available:

    default (default value)

    Utilization values are not considered. Resources are allocated according to location scoring. If scores are equal, resources are evenly distributed across nodes.

    utilization

    Utilization values are considered when deciding if a node has enough free capacity to satisfy a resource's requirements. However, load-balancing is still done based on the number of resources allocated to a node.

    minimal

    Utilization values are considered when deciding if a node has enough free capacity to satisfy a resource's requirements. An attempt is made to concentrate the resources on as few nodes as possible (to achieve power savings on the remaining nodes).

    balanced

    Utilization values are considered when deciding if a node has enough free capacity to satisfy a resource's requirements. An attempt is made to distribute the resources evenly, thus optimizing resource performance.

    NOTE: Configuring Resource Priorities

    The available placement strategies are best-effort—they do not yet use complex heuristic solvers to always reach optimum allocation results. Ensure that resource priorities are properly set so that your most important resources are scheduled first.

  5. Commit your changes before leaving crmsh:

    crm(live)configure# commit

The following example demonstrates a three node cluster of equal nodes, with 4 virtual machines:

crm(live)configure# node alice utilization memory="4000"
crm(live)configure# node bob utilization memory="4000"
crm(live)configure# node charly utilization memory="4000"
crm(live)configure# primitive xenA ocf:heartbeat:Xen \
    utilization hv_memory="3500" meta priority="10" \
    params xmfile="/etc/xen/shared-vm/vm1"
crm(live)configure# primitive xenB ocf:heartbeat:Xen \
    utilization hv_memory="2000" meta priority="1" \
    params xmfile="/etc/xen/shared-vm/vm2"
crm(live)configure# primitive xenC ocf:heartbeat:Xen \
    utilization hv_memory="2000" meta priority="1" \
    params xmfile="/etc/xen/shared-vm/vm3"
crm(live)configure# primitive xenD ocf:heartbeat:Xen \
    utilization hv_memory="1000" meta priority="5" \
    params xmfile="/etc/xen/shared-vm/vm4"
crm(live)configure# property placement-strategy="minimal"

With all three nodes up, xenA will be placed onto a node first, followed by xenD. xenB and xenC would either be allocated together or one of them with xenD.

If one node failed, too little total memory would be available to host them all. xenA would be ensured to be allocated, as would xenD. However, only one of xenB or xenC could still be placed, and since their priority is equal, the result is not defined yet. To resolve this ambiguity as well, you would need to set a higher priority for either one.

7.3.8 Configuring Resource Monitoring

To monitor a resource, there are two possibilities: either define a monitor operation with the op keyword or use the monitor command. The following example configures an Apache resource and monitors it every 60 seconds with the op keyword:

crm(live)configure# primitive apache apache \
  params ... \
  op monitor interval=60s timeout=30s

The same can be done with:

crm(live)configure# primitive apache apache \
   params ...
crm(live)configure# monitor apache 60s:30s

For an overview, refer to Section 4.3, Resource Monitoring.

7.3.9 Configuring a Cluster Resource Group

One of the most common elements of a cluster is a set of resources that needs to be located together. Start sequentially and stop in the reverse order. To simplify this configuration we support the concept of groups. The following example creates two primitives (an IP address and an e-mail resource):

  1. Run the crm command as system administrator. The prompt changes to crm(live).

  2. Configure the primitives:

    crm(live)# configure
    crm(live)configure# primitive Public-IP ocf:IPaddr:heartbeat \
       params ip=1.2.3.4 id=p.public-ip
    crm(live)configure# primitive Email lsb:exim \
       params id=p.lsb-exim
  3. Group the primitives with their relevant identifiers in the correct order:

    crm(live)configure# group g-shortcut Public-IP Email

To change the order of a group member, use the modgroup command from the configure subcommand. Use the following commands to move the primitive Email before Public-IP. (This is just to demonstrate the feature):

crm(live)configure# modgroup g-shortcut add p.lsb-exim before p.public-ip

In case you want to remove a resource from a group (for example, Email), use this command:

crm(live)configure# modgroup g-shortcut remove p.lsb-exim

For an overview, refer to Groups.

7.3.10 Configuring a Clone Resource

Clones were initially conceived as a convenient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing. They have turned out to be useful for several other purposes, including integrating with DLM, the fencing subsystem and OCFS2. You can clone any resource, provided the resource agent supports it.

Learn more about cloned resources in Clones.

Creating Anonymous Clone Resources

To create an anonymous clone resource, first create a primitive resource and then refer to it with the clone command. Do the following:

  1. Log in as root and start the crm interactive shell:

    root # crm configure
  2. Configure the primitive, for example:

    crm(live)configure# primitive Apache lsb:apache
  3. Clone the primitive:

    crm(live)configure# clone cl-apache Apache 

Creating Stateful/Multi-State Clone Resources

To create a stateful clone resource, first create a primitive resource and then the multi-state resource. The multi-state resource must support at least promote and demote operations.

  1. Log in as root and start the crm interactive shell:

    root # crm configure
  2. Configure the primitive. Change the intervals if needed:

    crm(live)configure# primitive my-rsc ocf:myCorp:myAppl \
        op monitor interval=60 \
        op monitor interval=61 role=Master
  3. Create the multi-state resource:

    crm(live)configure# ms ms-rsc my-rsc