Updating salt ssh clients start to fail with: IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $

This document (000021352) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Manager Server 4.3

Situation

After some time working normally, suddenly for example when attempting to apply updates to certain salt ssh clients, this fails with an error, excerpt:
 
Error calling Salt: com.suse.salt.netapi.exception.SaltException: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ 

Resolution

Remove the locally deployed salt bundle on the problem client by deleting the "/var/tmp/venv-salt-minion/" directory. After this, any action taken / executed against the client should re-deploy the salt bundle to the client.

Also consider adjusting the configuration for the "systemd-tmpfiles-clean.timer" to not cleanup files under "/var/tmp/".

Cause

The "systemd-tmpfiles-clean.timer" was configured to remove files under "/var/tmp" after a certain amount of time, resulting in a corrupt salt bundle installation on the client.

Additional Information

With salt ssh minions, the salt bundle is deployed by the salt-ssh pre-flight script to "/var/tmp/venv-salt-minion/". When the systemd cleanup timer runs and removes files older than the configured amount of time, this can corrupt the locally deployed salt bundle.

In this case the Red Hat minion had the file "/usr/lib/tmpfiles.d/tmp.conf" with the below configuration, that removed files older than 30 days from the "/var/tmp" directory, excerpt:
 
q /var/tmp 1777 root root 30d

To disable time based cleanup of the "/var/tmp" directory, using the previous example, adjust the line "/usr/lib/tmpfiles.d/tmp.conf" for example as follows:
 
q /var/tmp 1777 root root -
 
NOTE: The configuration files used by the "systemd-tmpfiles-clean.timer" may differ based on distribution. For example on newer SLES based clients the configuration for "/var/tmp" is located in "/usr/lib/tmpfiles.d/fs-var-tmp.conf" and by default does not have a time based cleanup configured for this directory.

For more information see "man 5 tmpfiles.d".

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000021352
  • Creation Date: 09-Feb-2024
  • Modified Date:09-Feb-2024
    • SUSE Manager

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center