"ceph-salt apply" fails with a connection error

This document (000020715) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7.1

Situation

When upgrading from SES 6 to SES 7.1 and applying the ceph-salt configuration as part of step 7. in the Deployment Guide this fails with the following error message for one or more hosts:

Error EINVAL: Failed to connect to <host> (<IP>)
Please make sure that the host is reachable and accepts connections using the cephadm SSH key

Also see TID 000020339 - ceph-salt apply fails with a timeout for "find an admin host"

Resolution

Add the cephadm group to the list of allowed groups in the ssh config (/etc/ssh/sshd_config) of the host by adding it to the "AllowGroups" line then restart the sshd service (systemctl restart sshd.service).

Cause

The cephadm user used for ssh by ceph-salt is not part of any of the allowed groups in the ssh configuration of the host(s).

Additional Information

Looking at the journalctl output for the sshd service (journalctl -u sshd.service) the following errors will be logged:

Jul 28 10:06:50 <hostname> sshd[1195888]: Postponed keyboard-interactive for invalid user cephadm from <host_ip_addr> port 36722 ssh2 [preauth]
Jul 28 10:06:50 <hostname> sshd[1195890]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<host_ip_addr> user=cephadm
Jul 28 10:06:52 <hostname> sshd[1195888]: error: PAM: Authentication failure for illegal user cephadm from <host_ip_addr>
Jul 28 10:06:52 <hostname> sshd[1195888]: Failed keyboard-interactive/pam for invalid user cephadm from <host_ip_addr> port 36722 ssh2
Jul 28 10:06:52 <hostname> sshd[1195888]: Postponed keyboard-interactive for invalid user cephadm from <host_ip_addr> port 36722 ssh2 [preauth]

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.