High sync times between SUSE Manager and SUSE Manager Proxy over WAN connections

Depending on what changes are executed in the WebUI or via an API call to distributions or auto-installation profiles, a cobbler sync command is required to transfer files from the SUSE Manager Server to the SUSE Manager Proxy systems. The SUSE Manager Proxies that should receive updates from cobbler are specified in /etc/cobbler/settings.

It is noticed, this process of syncing could take a considerable amount of time. As an example, looking at the logs in /var/log/cobbler/ the process started

Thu Jun  3 14:47:35 2021 - DEBUG | running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
Thu Jun  3 14:47:35 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*

and ends

Thu Jun  3 15:18:49 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/post/*
Thu Jun  3 15:18:49 2021 - DEBUG | shell triggers finished successfully

The transfer amount was roughly 1.8GB and took almost 30 minutes. 
Copying a single big file of the same size as /srv/tftboot is transferred within several minutes. 

Switching to an rsync based approach to copy files between SUSE Manager and SUSE Manager Proxy may help to reduce the transfer and wait times. A shell script to accomplish the task has been created and is available for download.

DISCLAIMER: The script and comments inside should provide a good overview of the process and what steps need to be taken into consideration. This script is NOT supported by SUSE. If help is required, please get in contact with SUSE Consulting to have the script adjusted for the respective environment.
 

Before using the script, please make sure any proxy in /etc/cobbler/settings has been disabled like:
#proxies:
# - "sumaproxy.sumaproxy.test"
# - "sumaproxy2.sumaproxy.test"

else SUSE Manager will continue to sync content to the proxies. Please restart the cobbler service to activate the changes:
systemctl restart cobblerd

For this issue to experience the following conditions must met:
 

1. /srv/tftboot contains a high number of files for distributions and client PXE boot files, in total several thousand files.
2. SUSE Manager Proxy systems are connected via a WAN connection

How to analyze:

As a first step it is required to know whether all involved proxy systems have a similar transfer rate and if the connection to a specific proxy is impacting the overall process. For this to know, please disable all but one proxy in /etc/cobbler/settings and run:

systemctl restart cobblerd
rm /var/lib/cobbler/pxe_cache.json
time cobbler sync

Repeat this procedure for every proxy to identify the SUSE Manager Proxies with the highest transfer time

The next steps would be:

1. Take a TCPdump between SUSE Manager and the involved systems:

On SUSE Manager:
tcpdump -i ethX -s 200 host <ip-address-of-susemanagerproxy> and not ssh

On SUSE Manager Proxy:
tcpdump -i ethX -s 200 host <ip-address-of-susemanager> and not ssh

This will only capture a package size of 200 which is sufficient to run an analysis, please adjust ethX to the respective network interface SUSE Manager uses to communicate with the proxy. At last, ssh communication will not be captured to reduce the number of packages even further. 

2. Start a cobbler sync. To force a sync, delete the cobbler json cache file first and then issue cobbler sync:

rm /var/lib/cobbler/pxe_cache.json
cobbler sync

3. When cobbler finished, stop the TCPdumps

4. Open the TCPdumps using Wireshark, go to Statistics -> Conversations and wait for the dump to be analyzed.

5. Switch to the TCP tab, the number shown on this tab gives the total number of conversations captured between SUSE Manager and SUSE Manager Proxy. 

6. Of interest is the column Duration, first sort ascending to know the minimal amount of time it took to transfer a file, then sort descending to know the max values for the big files like kernel and initrd transfers.
Please ignore packages on port 4505 and 4506 as these are used for salt communication. 

The analysis in this case showed the transfer of a ~1800bytes sized file took around 0.3 seconds from SUSE Manager to SUSE Manager Proxy. While there were not many big files, the high amount of small results in establishing a new connection for every file to be transferred. Knowing the minimal mount of time and number of connections allows (~5000 in this case) a rough estimation for the overall transfer time: 5000 * 0.3 / 60 = 25 minutes.