Running "ceph-salt update" fails with a python stack trace: KeyError

This document (000020413) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

When running "ceph-salt update" the command fails for a node with the following error, excerpt:
 
SLS: ceph-salt.common.sshkey
State: pkg.installed
Error Description:
An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 1999, in call
    )
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2036, in wrapper
    # improperly loaded
  File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 1690, in installed
    pkgs, refresh = _resolve_capabilities(pkgs, refresh=refresh, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 1004, in _resolve_capabilities
    ret = __salt__["pkg.resolve_capabilities"](pkgs, refresh=refresh, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 3130, in resolve_capabilities
    refresh_db(root)
  File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 1440, in refresh_db
    out = __zypper__(root=root).refreshable.call(*refresh_opts)
  File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 381, in __call
    self.__call_result = __salt__["cmd.run_all"](cmd, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1269, in __getitem__
  File "/usr/lib/python3.6/site-packages/salt/utils/lazy.py", line 111, in __getitem__
KeyError: 'cmd.run_all'

Resolution

The salt minion service needs to be properly started / restarted using systemd.

Cause

The salt minion service was manually started outside of systemd / not using the systemd service at an earlier point in time using an out dated python / salt version.

Additional Information

The "salt-minion.service" shows the service as "Active: inactive (dead)", however, there are active salt processes on the host listed when running for example:
 
ps aux | grep salt-minion
root        3070  0.0  0.1 173948 29980 ?        Ss   Oct01   0:00 /usr/bin/python3 /usr/bin/salt-minion

These processes were started before installing updates that included python and salt updates, since they were started manually, the processes were not restarted automatically after the updates were applied.

It can also be useful to make sure all modules are synced to the minion(s) by on the admin node running:
 
salt '*' saltutil.sync_all

 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020413
  • Creation Date: 21-Oct-2021
  • Modified Date:21-Oct-2021
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center