High sync times between SUSE Manager and SUSE Manager Proxy over WAN connections

This document (000020573) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Manager 4.x
SUSE Manager Proxy 4.x

Situation

Depending on what changes are executed in the WebUI or via an API call to distributions or auto-installation profiles, a cobbler sync command is required to transfer files from the SUSE Manager Server to the SUSE Manager Proxy systems. The SUSE Manager Proxies that should receive updates from cobbler are specified in /etc/cobbler/settings.

It is noticed, this process of syncing could take a considerable amount of time. As an example, looking at the logs in /var/log/cobbler/ the process started

Thu Jun  3 14:47:35 2021 - DEBUG | running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
Thu Jun  3 14:47:35 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*


and ends

Thu Jun  3 15:18:49 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/post/*
Thu Jun  3 15:18:49 2021 - DEBUG | shell triggers finished successfully


The transfer amount was roughly 1.8GB and took almost 30 minutes. 
Copying a single big file of the same size as /srv/tftboot is transferred within several minutes. 

Resolution

Switching to an rsync based approach to copy files between SUSE Manager and SUSE Manager Proxy may help to reduce the transfer and wait times. A shell script to accomplish the task has been created and is available for download.

DISCLAIMER: The script and comments inside should provide a good overview of the process and what steps need to be taken into consideration. This script is NOT supported by SUSE. If help is required, please get in contact with SUSE Consulting to have the script adjusted for the respective environment.
 

Before using the script, please make sure any proxy in /etc/cobbler/settings has been disabled like:
#proxies:
# - "sumaproxy.sumaproxy.test"
# - "sumaproxy2.sumaproxy.test"


else SUSE Manager will continue to sync content to the proxies. Please restart the cobbler service to activate the changes:
systemctl restart cobblerd

Cause

For this issue to experience the following conditions must met:
 

  1. /srv/tftboot contains a high number of files for distributions and client PXE boot files, in total several thousand files.
  2. SUSE Manager Proxy systems are connected via a WAN connection

The current design in SUSE Manager 4.x, cobbler is not able to sync only changed/recently added files. As a consequence, an execution of cobbler sync triggers a full sync of the /srv/tftpboot directory to all specified proxies configured in /etc/cobbler/settings and is influenced by the latency of the WAN connection between the involved systems. 

Additional Information

How to analyze:

As a first step it is required to know whether all involved proxy systems have a similar transfer rate and if the connection to a specific proxy is impacting the overall process. For this to know, please disable all but one proxy in /etc/cobbler/settings and run:

systemctl restart cobblerd
rm /var/lib/cobbler/pxe_cache.json
time cobbler sync


Repeat this procedure for every proxy to identify the SUSE Manager Proxies with the highest transfer time

The next steps would be:

1. Take a TCPdump between SUSE Manager and the involved systems:

On SUSE Manager:
tcpdump -i ethX -s 200 host <ip-address-of-susemanagerproxy> and not ssh

On SUSE Manager Proxy:
tcpdump -i ethX -s 200 host <ip-address-of-susemanager> and not ssh

 

This will only capture a package size of 200 which is sufficient to run an analysis, please adjust ethX to the respective network interface SUSE Manager uses to communicate with the proxy. At last, ssh communication will not be captured to reduce the number of packages even further. 


2. Start a cobbler sync. To force a sync, delete the cobbler json cache file first and then issue cobbler sync:

rm /var/lib/cobbler/pxe_cache.json
cobbler sync


3. When cobbler finished, stop the TCPdumps

4. Open the TCPdumps using Wireshark, go to Statistics -> Conversations and wait for the dump to be analyzed.

5. Switch to the TCP tab, the number shown on this tab gives the total number of conversations captured between SUSE Manager and SUSE Manager Proxy. 

6. Of interest is the column Duration, first sort ascending to know the minimal amount of time it took to transfer a file, then sort descending to know the max values for the big files like kernel and initrd transfers.
Please ignore packages on port 4505 and 4506 as these are used for salt communication. 

The analysis in this case showed the transfer of a ~1800bytes sized file took around 0.3 seconds from SUSE Manager to SUSE Manager Proxy. While there were not many big files, the high amount of small results in establishing a new connection for every file to be transferred. Knowing the minimal mount of time and number of connections allows (~5000 in this case) a rough estimation for the overall transfer time: 5000 * 0.3 / 60 = 25 minutes. 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020573
  • Creation Date: 01-Feb-2022
  • Modified Date:10-Feb-2022
    • SUSE Manager Server
    • SUSE Manager Proxy

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center