Images not syncing in a slow environment

This document (000020522) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Manager for Retail 4.2
SUSE Manager for Retail 4.1

Situation

In a SUSE Manager for Retail environment, images fail to download, showing the following error (adapt hostnames and image names):

Comment: An exception occurred in this state: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/state.py", line 2187, in call *cdata["args"], **cdata["kwargs"] File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2113, in wrapper return f(*args, **kwargs) File "/var/cache/salt/minion/extmods/states/image_sync.py", line 70, in image_synced bundle = _download_bundle(image_data) File "/var/cache/salt/minion/extmods/states/image_sync.py", line 37, in _download_bundle return __salt__['cp.get_url'](image_data['sync']['bundle_url'], source_hash=image_data['sync'].get('bundle_hash')) File "/usr/lib/python3.6/site-packages/salt/modules/cp.py", line 388, in get_url path, dest, makedirs, saltenv, source_hash=source_hash File "/usr/lib/python3.6/site-packages/salt/fileclient.py", line 766, in get_url raise MinionError("Error: {} reading {}".format(query["error"], url)) salt.exceptions.MinionError: Error: HTTP 599: Timeout during request reading https://SUMA.fqdn.com/os-images/1//imagename.tgz

This might be observed specially when the image grows in size, which can happen when adding modules to it.

Resolution

The following salt state helps modifying and applying a reconfiguration of the salt minion:

/etc/salt/minion.d/http_timeout.conf:
file.managed:
- contents:
- "http_request_timeout: 4900.0"

restart_salt_minion:
cmd.run:
    - name: sleep 5 && systemctl restart salt-minion
    - bg: True
    - require:
      - file: /etc/salt/minion.d/http_timeout.conf

The file should be saved as /srv/salt/set_connection_timeout.sls (or any other convenient name) on the salt master (SUSE Manager). This will add the configuration file to the minion configuration and then only a restart of the salt minion would be needed.

The state can be easily applied to the branch server (SUSE Manager Proxy) by running from the master:

salt $branchserver state.apply set_connection_timeout

As commented before, a restart of the minion would be needed (to be run from the branch server, which is a SUSE Manager Proxy):

systemctl restart salt-minion.service

Cause

Maximum download time for image sync process is 1 hour per image download by default.
In case of large images and/or slow network, the parameter http_request_timeout should be increased.
As this needs to be done on the minion, a salt state is the most convenient approach.
In this example the parameter was increased to 4900 (seconds), but obviously the value to avoid a timeout and solve the issue can be adjusted as needed.

Status

Reported to Engineering

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.