High sync times between SUSE Manager and SUSE Manager Proxy over WAN connections
This document (000020573) is provided subject to the disclaimer at the end of this document.
SUSE Manager Proxy 4.x
Depending on what changes are executed in the WebUI or via an API call to distributions or auto-installation profiles, a cobbler sync command is required to transfer files from the SUSE Manager Server to the SUSE Manager Proxy systems. The SUSE Manager Proxies that should receive updates from cobbler are specified in /etc/cobbler/settings.
It is noticed, this process of syncing could take a considerable amount of time. As an example, looking at the logs in /var/log/cobbler/ the process started
Thu Jun 3 14:47:35 2021 - DEBUG | running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
Thu Jun 3 14:47:35 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
Thu Jun 3 15:18:49 2021 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/task/sync/post/*
Thu Jun 3 15:18:49 2021 - DEBUG | shell triggers finished successfully
The transfer amount was roughly 1.8GB and took almost 30 minutes.
Copying a single big file of the same size as /srv/tftboot is transferred within several minutes.
Switching to an rsync based approach to copy files between SUSE Manager and SUSE Manager Proxy may help to reduce the transfer and wait times. A shell script to accomplish the task has been created and is available for download.
DISCLAIMER: The script and comments inside should provide a good overview of the process and what steps need to be taken into consideration. This script is NOT supported by SUSE. If help is required, please get in contact with SUSE Consulting to have the script adjusted for the respective environment.
Before using the script, please make sure any proxy in /etc/cobbler/settings has been disabled like:
# - "sumaproxy.sumaproxy.test"
# - "sumaproxy2.sumaproxy.test"
else SUSE Manager will continue to sync content to the proxies. Please restart the cobbler service to activate the changes:
systemctl restart cobblerd
For this issue to experience the following conditions must met:
- /srv/tftboot contains a high number of files for distributions and client PXE boot files, in total several thousand files.
- SUSE Manager Proxy systems are connected via a WAN connection
The current design in SUSE Manager 4.x, cobbler is not able to sync only changed/recently added files. As a consequence, an execution of cobbler sync triggers a full sync of the /srv/tftpboot directory to all specified proxies configured in /etc/cobbler/settings and is influenced by the latency of the WAN connection between the involved systems.
How to analyze:
As a first step it is required to know whether all involved proxy systems have a similar transfer rate and if the connection to a specific proxy is impacting the overall process. For this to know, please disable all but one proxy in /etc/cobbler/settings and run:
systemctl restart cobblerd
time cobbler sync
Repeat this procedure for every proxy to identify the SUSE Manager Proxies with the highest transfer time
The next steps would be:
1. Take a TCPdump between SUSE Manager and the involved systems:
On SUSE Manager:
tcpdump -i ethX -s 200 host <ip-address-of-susemanagerproxy> and not ssh
On SUSE Manager Proxy:
tcpdump -i ethX -s 200 host <ip-address-of-susemanager> and not ssh
This will only capture a package size of 200 which is sufficient to run an analysis, please adjust ethX to the respective network interface SUSE Manager uses to communicate with the proxy. At last, ssh communication will not be captured to reduce the number of packages even further.
2. Start a cobbler sync. To force a sync, delete the cobbler json cache file first and then issue cobbler sync:
3. When cobbler finished, stop the TCPdumps
4. Open the TCPdumps using Wireshark, go to Statistics -> Conversations and wait for the dump to be analyzed.
5. Switch to the TCP tab, the number shown on this tab gives the total number of conversations captured between SUSE Manager and SUSE Manager Proxy.
6. Of interest is the column Duration, first sort ascending to know the minimal amount of time it took to transfer a file, then sort descending to know the max values for the big files like kernel and initrd transfers.
Please ignore packages on port 4505 and 4506 as these are used for salt communication.
The analysis in this case showed the transfer of a ~1800bytes sized file took around 0.3 seconds from SUSE Manager to SUSE Manager Proxy. While there were not many big files, the high amount of small results in establishing a new connection for every file to be transferred. Knowing the minimal mount of time and number of connections allows (~5000 in this case) a rough estimation for the overall transfer time: 5000 * 0.3 / 60 = 25 minutes.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020573
- Creation Date: 01-Feb-2022
- Modified Date:10-Feb-2022
- SUSE Manager Server
- SUSE Manager Proxy
For questions or concerns with the SUSE Knowledgebase please contact: firstname.lastname@example.org