Proactive patch management & compliance: Hardening the hybrid Azure fleet at scale

Share
Share

Welcome back to 

SUSE Solutions on Azure: The Technical Series.

Bridging the Gap Between Linux Freedom and Azure Scale

Enterprise Linux on Azure requires a careful balance between open source flexibility and corporate security control. This series provides technical blueprints to help you shift away from managing individual distributions. Learn to build a unified, secure and resilient Linux stack at any scale.

Part 3: Proactive patch management & compliance: Hardening the hybrid Azure fleet at scale

With your fleet systematically reporting back to the management plane (Part 2), day-to-day operations shift toward risk mitigation. 

This part establishes an automated pipeline. It tracks, audits and remediates software flaws and configuration deviations across Linux distributions on Azure.

Key Takeaways: What you will accomplish

  • Automated Patching:Sync CVE metadata and manage patch remediation directly from the SMLM UI.
  • System Auditing:Deploy OpenSCAP tools to scan and audit configurations across diverse Linux distributions.
  • Compliance Automation:Map, schedule and enforce vendor-standard security profiles systematically.
  • Streamlined Remediation:Utilize structured system group definitions to safely execute compliance-as-code.
  • Verification:Track compliance states and generate detailed HTML reports for audit validation.

 

Architectural introduction and prerequisites

Native vulnerability tracking vs. intrusive scanning

A primary issue when managing a cloud footprint is the performance overhead introduced by security utilities. In SUSE Multi-Linux Manager (SMLM), there is a distinct difference between scanning for missing patches (CVEs) and scanning for configuration compliance (OpenSCAP).

  1. Native CVE Auditing (For Missing Security Patches): Rather than running heavy network discovery tools that execute resource-intensive, brute-force scans against production servers, SMLM relies on your configured software channels.
    It cross-references the software packages actively installed on your clients against patch metadata synced directly to the server. Finding a missing security patch becomes an instantaneous database lookup requiring no client-side action.
  2. Deep Configuration Auditing (OpenSCAP): For stricter security boundaries, SMLM uses the Security Content Automation Protocol (SCAP) to evaluate explicit operating system file permissions, kernel configuration values and user access policies against certified baselines.

 

A CVE (Common Vulnerabilities and Exposures) is a fix for a publicly known security vulnerability. Each CVE contains an identification number, a description of the vulnerability and links to further information. CVE identification numbers use the form CVE-YEAR-XXXX

SCAP stands for Security Content Automation Protocol. It is a framework of specifications that support automated configuration, vulnerability scanning and policy compliance evaluation of systems deployed in an organization.

OpenSCAP is a collection of open source tools implementing the SCAP framework for Linux. The National Institute of Standards and Technology (NIST) maintains SCAP.

To harden a system, OpenSCAP uses security rules that define certain security measures. Multiple rules can be combined into profiles. See the OpenSCAP documentation for details.

 

Decoupling scale barriers: The Salt ZeroMQ bus vs. agentless SSH

Many cloud organizations have standardized on agentless, SSH-driven configuration tools (like standard Ansible Automation Platform structures). Marketing often frames “agentless” operations as an advantage because they eliminate client-side tools. However, they introduce severe architectural bottlenecks when patching and auditing at cloud scale.

Scaling fleets into thousands of instances creates intense CPU encryption overhead on the management node. It triggers frequent connection timeouts and leaves inbound port 22 (SSH) open across production subnets – violating zero trust security perimeters.

We deployed a lightweight local daemon (venv-salt-minion) that maintains a secure, persistent outbound-only connection to the SMLM private IP during the onboarding.

Instead of looping over SSH, SMLM broadcasts execution metadata simultaneously across an asynchronous ZeroMQ message bus. This architecture handles distinct command tracking functions via explicit network paths.

This communication standard segments management traffic across distinct target ports:

  • Port 443 (HTTPS): Handles web UI delivery, internal API communications, package mirror downloads and telemetry payload delivery.
  • Port 4505 (TCP): Functions as the Salt Master “Publisher” lane to track fleet connection presence and broadcast commands.
  • Port 4506 (TCP): Functions as the Salt Master “Request Server” lane to securely accept execution returns from the fleet.

When an administrator schedules a patch or a compliance audit, the directive is broadcast over the message bus. Thousands of nodes ingest the payload simultaneously, process the package logic locally and asynchronously and stream structured results back via the request lane, remediating vulnerabilities across your infrastructure in seconds.

 

Enterprise architectural constraints and perimeter protection

Before we execute remediation campaigns, as a short recap, your environment must satisfy three mandatory security perimeters:

  • Network Isolation: Ensure your Azure Network Security Groups (NSGs) permit traffic along these management lines. Apply explicit inbound firewall rules to the SMLM server private IP to allow ports 443, 4505 and 4506 from the client fleet subnet. If these ports are obstructed, clients will register but remain completely “Unreachable” within the console.
  • Name Resolution: Bidirectional Fully Qualified Domain Name (FQDN) resolution is required across all nodes to prevent unpredictable system component failure. If you do not have an own domain and DNS, link an Azure Private DNS Zone (e.g., mlm.azure.local) with auto-registration active to your virtual networks to guarantee hostnames resolve dynamically across isolated virtual network boundaries.
  • Perimeter Protection: Default public access endpoints on your management server must be strictly limited. Limit, filter and monitor all administrative entry point networks. Production setups should leverage an Azure Bastion host or an encapsulated private network architecture rather than exposing public IPs to internet-facing threat actors.

 

Prerequisites for patch metadata and scanner 

The SMLM server utilizes a scheduled internal background service to aggregate and format upstream CVE security advisories into local query tables. 

For SCAP scanning it’s a bit more complex as the clients need the scanning engine and corresponding SCAP Content package installed. Depending on the operating system, these packages are included either on the base operating system or SUSE Manager Client Tools.

As different operating systems make available different OpenSCAP scanner packages, content files and profiles, you need to look once at a representative client for the right package. 

SUSE provides the scap-security-guide package for different openscap profiles. 

In the current version of scap-security-guide, SUSE supports the following profiles:

  • DISA STIG profile for SUSE Linux Enterprise Server
  • ANSSI-BP-028 profile for SUSE Linux Enterprise Server
  • PCI-DSS profile for SUSE Linux Enterprise Server
  • HIPAA profile for SUSE Linux Enterprise Server
  • Hardening for Public Cloud Image of SUSE Linux Enterprise Server for SAP Applications 
  • Public Cloud Hardening for SUSE Linux Enterprise
  • Standard System Security profile
  • CIS profile for SUSE Linux Enterprise Server


The CIS profile is community supplied and not officially supported by SUSE.

For Non-SUSE operating systems the included profiles are community supplied, therefore they are not officially supported by SUSE.

As one content file may contain more than one profile you need to verify the exact profile definitions embedded within the vendor content files.

Cross-distribution package and directory layout matrix

Use the following reference matrix to identify your foundational landscape components

Linux Distribution Family Scanner Package Security Content Package Content Directory
SUSE Linux Enterprise openscap-utils scap-security-guide /usr/share/xml/scap/ssg/content/
Red Hat Enterprise based openscap-utils scap-security-guide-redhat /usr/share/xml/scap/ssg/content/
Ubuntu Server libopenscap8 scap-security-guide-ubuntu /usr/share/xml/scap/ssg/content/
Debian Linux libopenscap8 scap-security-guide-debian /usr/share/xml/scap/ssg/content/

On a representative target client node, install the Security Content Package to be able to find the available profiles.

  1. Login as admin of such a representative client.
  2. Determine the location of the available SCAP files in the Content Directory, e.g.
    ls -l /usr/share/xml/scap/ssg/content
  3. Pick the file that suits your needs, for example we use the ssg-sle15-ds.xml and to list the available profiles within this file use.

oscap info /usr/share/xml/scap/ssg/content/ssg-sle15-ds.xml (1)

This will give you the target baseline profiles and the IDs

Note the file path (1) and profile (2) to perform the scan later.

In the example above we use the file /usr/share/xml/scap/ssg/content/ssg-sle15-ds.xml and choose the STIG profile, so the Id: xccdf_org.ssgproject.content_profile_stig

Note: SLES on Azure uses the kernel-azure by default. Some profiles are not aware of this kernel and may provide wrong information. If you want to use such profiles you need to switch to kernel-default.

 

Step-by-step implementation

Before conducting an evaluation, the server’s cache must reflect the latest vulnerability indexes. Then, you can run a live vulnerability audit to identify and patch unpatched hosts directly from the SMLM graphical interface.

 

Note: While this service executes automatically every evening at 23:00, manual execution is recommended to ensure you have the latest data before you conduct a fleet audit.

Synchronize upstream CVE vulnerability data

  • Log into the SMLM Web UI and navigate to Admin > Task Schedules in the left navigation panel.
  • Locate the internal processing bunch designated as cve-server-channels-bunch.
  • Click on the link cve-server-channels-bunch and click on Single Run Schedule on the right to trigger an immediate, manual metadata compilation.

Audit the cloud fleet for security flaws:

  • Navigate to Audit > CVE Audit in the primary navigation menu.
  • Locate the CVE Identifier search input field. Type the targeted tracking identifier (e.g., CVE-2026-1502) into the search block and click Audit Servers.
  • Review the status matrix outputs instantly populated via the database cross-reference like in the example below:

You have now several ways to patch the affected systems

  • On the results screen, select the checkbox next to the host systems flagged with the Red or Yellow Shield. This adds the chosen system to the System Set Manager (SSM) at the right top of the screen (1 system selected).

The system set manager (SSM) is used to perform actions on more than one client at a time. SSM creates ephemeral sets of clients, making it useful for one-off actions that you need to apply to a number of clients. 

Navigate directly to the Systems > System Set Manager workspace and select the Patches tab.

  • Alternatively, click the link in the Actions tab “Install a new patch on this system” which will bring you to the software tab of the chosen machine. You will see all relevant patches for this machine.
  • If you click the patch name instead (e.g. SUSE-15-SP7-2026-1715). It will bring you to the Security Advisory Screen where you can see Detail, the packages and affected systems.

After clicking Apply Patches, SMLM schedules the installation via the outbound Salt layer, upgrading packages simultaneously without requiring SSH configurations.

Patches Available:

The menu Patches helps you find and manage available patches for your client systems. 

A good overview is at Patches > PatchList > Relevant at the tab Security Patches. It gives you an overview of all patches applying to your client systems and shows the CVE numbers too.

This is the quick view of what is going on in your fleet.

Perform an OpenSCAP audit

Preparation 

Deploy Security Policy Assets to all your client systems:

  • Navigate to Systems and click on your target system name.
  • Go to Software > Packages > Install within the main menu.
  • Search for the scanner package and the definitive policy content guide package matching your target distribution (e.g., scap-security-guide and openscap-utils) and select the package with the highest version number by clicking into the checkbox. 
  • Click Install Selected Packages and confirm it to push the assets down to the systems.

Discover the Valid Policy Profile Identifiers as shown in earlier chapter

  • SSH into a representative target client node through your secure path (e.g. Bastion Host or similar).
  • Query the data stream profile identifiers structured inside the content XML files using the native tool matching your distribution:

Schedule the native OpenSCAP scan via the Web UI form:

  • Remember the values for the profile and the path to the XCCDF document 
  • Return to the SMLM Web UI, navigate to Systems and click directly on the name of your target client node.
  • Select the Audit tab from the node’s sub-navigation list and click OpenSCAP and then Schedule.
  • Fill out the integrated UI configuration form fields explicitly using the discovered paths and parameters :
    • For example, enter these two configuration values:
                    –profile xccdf_org.ssgproject.content_profile_stig
      /usr/share/xml/scap/ssg/content/ssg-sle15-ds.xml

  • Click Schedule. The scan runs automatically at the machine’s next scheduled check-in window.

 

Validation and health verification

To ensure that your vulnerability remediation and policy enforcement frameworks are working correctly, validate the outputs inside the compliance control dashboards.

Analyzing granular compliance reports and visual indicators

To review the complete evaluation results generated by your OpenSCAP form-based scans, ensure that detailed file logging is active at the tenant level:

  1. In the SMLM Web UI, navigate to Admin > Organizations and select the organization your client hosts belong to.
  2. Click the Configuration tab, locate the Enable Upload of Detailed SCAP Files option, check the box and save.

Once enabled, navigating to Audit > OpenSCAP > All Scans will display your historical evaluation summaries. Clicking on a  Xccdf Profile link row opens a fully itemized HTML report detailing every pass, fail, or error metric across your system configurations.

Navigate to System >  Audit>  OpenScap > List Scans

  • At the entry List Scans you will see the result of the scan and rough states. 
  • Click the result name to bring you to a more detailed page. A detailed report is available at the link report.html.
  • The log shows whether each test has passed, failed or remained unchecked. Clicking on a test provides more details about the test itself.

 

Simple and safe Multi-Linux remediation via SSM

When an OpenSCAP compliance audit flags system drift or failed validation rules, you must deploy a remediation policy to bring your infrastructure back into line. The OpenSCAP package scap-security-guide provides built-in compliance remediation Bash scripts and Ansible playbooks inside the same security guide content packages.

To eliminate error-prone scripting variables, the standard administrative best practice is to group your systems by distribution family and execute a clean, direct task through the System Set Manager (SSM).

Step-by-step UI execution pattern

  1. Target a Specific Distribution Group:
    • Navigate to System Groups inside the SMLM console and click Use in SSM next to your specific target pool (e.g., your homogeneous SLES15-Production group).
  1. Load the Direct Vendor Remediation Path:
    • Go to the System Set Manager (SSM) workspace menu and click Misc > Remote Command.
    • Ensure the active execution user field is set to root.
    • Enter a memorable Command label to simplify tracking within the execution log later, e.g “RemediationScript”
    • Provide the script, in our example for SLES15. Adapt it for other operating systems and versions.

  • Click Schedule.

By breaking your remediation tasks into clear, OS-specific system groups, you eliminate the risks of multi-OS conditional programming within a web interface text block. SMLM safely pushes the execution directive across the outbound message bus and applies the policy vendor-side.

Brief note on multi-engine flexibility

If your team handles configuration management exclusively through other internal standards, SMLM can alternatively apply these remediations through alternate pipelines:

  • Declarative Salt States (.sls): You can bind the remediation file script directly to SMLM’s recurring configuration channels to create an automatic, self-healing loop that rolls back drift automatically without manual clicks.
  • Ansible Integration: Because the security guide package also contains matching Ansible definitions (e.g., sle15-playbook-stig.yml), you can trigger these playbooks natively from the SMLM control console, leveraging your team’s Ansible automation standards over the underlying Salt transport architecture.

Get the log from the remediation script run

Go to System > Events Actions > History and click on the Action Link name we gave our run. In the example we had chosen “RemediationScript” we see the return state and the log output.

If you now do a second OpenSCAP Scan you can compare the runs to see what is still open and may need adoption. You can do this by customizing the existing catalogs using the SCAP Workbench (package scap-workbench) which is available in the repositories too. You can disable checks which may not be applicable for you or modify some thresholds.

Much more detailed information about the topic in general is available within the SUSE documentation at the Security and Hardening Guide for the relevant version.

If you are interested in the topic, the source of ready to use security guides is the open source project Compliance-as-code. You will find under content / products / <OS_VARIANT> / profiles many further pre-build profiles.

With the SMLM version 5.2 there will be several simplifications to all the shown steps. The general context is the same, but you have more simplification and integration in the web UI.

 

Evaluating Azure pre-hardened images post-boot delta

To streamline this verification workflow, SUSE provides pre-hardened images directly in the Azure Marketplace (e.g., SLES and SLES for SAP Applications). These images are structurally optimized using a combination of the STIG and CIS profiles (pcs-hardening.profile).

However, cloud lifecycle parameters dictate that certain security variables can only be safely evaluated or applied after an instance has booted post-deployment:

  • Root Account Passwords: Cannot be pre-configured during image generation without risking public secret exposure.
  • Network Parameters: Firewall access rules must adapt dynamically to your infrastructure framework during instance creation.
  • Storage Partitioning: Standard cloud frames use default disk maps; stricter directory separation requires post-boot configurations.
  • Package Exclusions: Because tenant application requirements vary wildly, it is impossible to determine generic package safety profiles beforehand.

Look for the additional “Hardened” in the title of the SUSE images, like

 

Concluding summary checklist

Infrastructure lifecycle checklist

By completing this installment, your hybrid environment has achieved the following technical milestones:

[ ] Synchronized the local SMLM vulnerability repository metadata cache with real-time upstream CVE advisory indexes.

[ ] Audited the environment for specific security flaws and remediated missing packages from a single UI interface.

[ ] Deployed cross-distribution OpenSCAP scanning instrumentation tools and security guides across active client endpoints.

[ ] Discovered, mapped and scheduled automated compliance evaluations using authoritative vendor profile identifiers.

[ ] Enforced Compliance-as-Code standards utilizing vendor-provided remediations and simple, clean target system group definitions.

 

What’s Next? To safely execute these discovered security patches and configuration remediations without risking production application downtime, you need a mechanism to isolate updates. In the next part we will look into: Content Lifecycle Management (CLM), we will explore how to freeze repository states and safely promote software channels through disciplined Dev, QA and Production environment staging.

Share
(Visited 2 times, 1 visits today)
Avatar photo
260 views