Collecting cluster report fails while required to use forwarded SSH agent

This document (000020662) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP4 / crmsh < 4.4.0+20220708.6ed6b56f-150400.3.3.1
SUSE Linux Enterprise Server for SAP Applications 15 SP4 / crmsh < 4.4.0+20220708.6ed6b56f-150400.3.3.1
SUSE Linux Enterprise Server 15 SP3 / crmsh < 4.3.1+20220610.733357e2-150200.5.83.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 / crmsh < 4.3.1+20220610.733357e2-150200.5.83.1
SUSE Linux Enterprise Server 15 SP2 / crmsh< 4.3.1+20220610.733357e2-150200.5.83.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 / crmsh< 4.3.1+20220610.733357e2-150200.5.83.1

Situation

Due to a restricted environment requirements sysadmins are required to do only SSH 'publickey' authentication method (ie. password interactive login is forbidden); forwarded SSH agent is also used when attempting to collect cluster report (eg. no SSH private keys are allowed to be located on the cluster nodes).

A sysadmin is using SSH agent and collection of cluster report fails

A sysadmin needs to collect cluster report from two nodes, he uses SSH agent forwarding as loading SSH private keys onto the systems is forbidden; the sysadmin expects SSH connection from one node to the other would use his SSH forwarded via SSH agent.

First see that public key authentication proxied via SSH agent works as expected:

# s153cl1 - main node
sadmin1@s153cl1:~> echo $SSH_AUTH_SOCK
/tmp/ssh-rDTCYLyHvd/agent.2865

# checking other node from main node
sadmin1@s153cl1:~> ssh -v s153cl2 hostname 2>&1 | \
    grep -P '(Server accepts key:|^s153)'
debug1: Server accepts key: sadmin1@workstation RSA SHA256:KMxvvfn9io9D1y/QY0tnJ4AxYKziX3F3G0oCrP3fFDA agent
s153cl2

Now see that similar sudo rules as added on both nodes:

$ sudo -l
Matching Defaults entries for sadmin1 on s153cl1:
    always_set_home, secure_path=/usr/sbin\:/usr/bin\:/sbin\:/bin\:/usr/local/bin\:/usr/local/sbin, env_reset,
    env_keep="LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME
    LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE", !insults

Runas and Command-specific defaults for sadmin1:
    Defaults!/usr/sbin/crm report * env_keep+=SSH_AUTH_SOCK

User sadmin1 may run the following commands on s153cl1:
    (root) NOPASSWD: /usr/sbin/crm report *

That is, the sudoers definition would be something like this (preserving SSH_AUTH_SOCK for users permitted to run crm report):

Host_Alias CLUSTER = s153cl1, s153cl2
Runas_Alias R = root
Defaults!HA_ALLOWED env_keep+=SSH_AUTH_SOCK
Cmnd_Alias HA_ALLOWED = /usr/sbin/crm report *

%sysadmins CLUSTER = (R) NOPASSWD: HA_ALLOWED

Finally, the attempt to collect cluster report while using SSH agent, it fails:

sadmin1@s153cl1:~> sudo /usr/sbin/crm report -u sadmin1
WARNING: s153cl1# could not figure out the log format of /var/log/cluster/corosync.log
WARNING: s153cl1# ERROR: s153cl2# cannot find pe daemon directory!
INFO: s153cl1# Trying connect by 192.168.122.12
WARNING: s153cl1# ERROR: s153cl2# cannot find pe daemon directory!

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/crmsh/hb_report/utillib.py", line 1551, in start_slave_collector
    crmutils.get_stdout(cmd, input_s=eval(compress_data))
  File "<string>", line 0
    
    ^
SyntaxError: unexpected EOF while parsing

Resolution

A sysadmin is using SSH agent and collection of cluster report fails

The failure was reported to engineering and solved in newer version of crmsh package. If the package is not updated to the version with fix, one can collect cluster report from single nodes, one by one.

ssh <user>@<cluster node> sudo -u root /usr/sbin/crm report -S /home/<user>/<cluster node>

And then download the cluster report archive from the nodes from /home/<user>/<cluster node>.tar.bz2.

Cause

A sysadmin is using SSH agent and collection of cluster report fails

The code was not working correctly with SSH agent, engineering was informed about the issue and new version of crmsh fixed it.

Status

Reported to Engineering

Additional Information

https://documentation.suse.com/sle-ha/15-SP3/html/SLE-HA-all/app-crmreport-nonroot.html

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

Document ID:000020662
Creation Date: 13-Oct-2022
Modified Date:14-Oct-2022
- SUSE Linux Enterprise High Availability Extension
- SUSE Linux Enterprise Server for SAP Applications

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com