IT services provider GESIS (Gesellschaft für Informationssysteme mit beschränkter Haftung) offers network, data center, e-business, business intelligence, document management and SAP application services to the Salzgitter Group steel business. Based in Salzgitter, Germany, the company employs 200 people.
With more than 30 years’ experience under its belt, GESIS offers solutions and services covering everything from infrastructure and operations, manufacturing, material logistics and production assurance to finance and controlling, marketing and sales, and human resources.
GESIS is part of Salzgitter Group, one of the largest steel producers in Europe with approximately 25,000 employees and annual sales of 9 billion euros. The group consists of more than 150 subsidiaries worldwide, and GESIS is the central IT services provider to all group companies.
At-a-Glance
With 24-hour steel manufacturing operations worldwide, there’s no room for downtime. To keep mission-critical business applications running securely and smoothly, GESIS deployed SUSE® Linux Enterprise Live Patching, enabling system administrators to apply essential Linux kernel security patches without having to take systems offline. Today, GESIS no longer needs to convince reluctant business leaders to schedule unwanted downtime windows, but can seamlessly patch systems every six to eight weeks without service disruption.
The Challenge
Enabling business growth
Global demand for steel is growing steadily. Despite a slowdown in some markets in recent years, most notably in China, demand for steel in emerging economies is soaring. Booming industry, increasing urbanization and ambitious infrastructure projects in countries such as Brazil and Russia all call for steel products — and Salzgitter Group is looking to capitalize on this demand.
To make the most of the current economic climate, Salzgitter has its sights firmly set on growth and innovation across all markets. As part of its forward-looking business strategy, the group is targeting organic growth of more than 250 million euros a year over the period from 2017 to 2021. With a global production and service network already in place, Salzgitter Group has a significant advantage over its competitors. The group has manufacturing sites all over the world, including Germany, the United States, Mexico, Brazil, India and China. These operations run 24 hours a day, 365 days a year. To keep production lines rolling and finished products on the move to customers without delays, the company depends on IT systems. In this highly automated and closely connected manufacturing and logistics environment, mission-critical SAP business applications are particularly important to ensure the business runs reliably day after day.
Thomas Lowin, server operations lead at GESIS, says: “While other companies can simply halt production lines, this is not possible with an integrated steel mill. To support 24/7 production, Salzgitter Group depends on the constant availability and robust service of its IT systems. Every minute of unplanned downtime would cost thousands of euros and jeopardize complex production schedules of diverse steel products.”
Balancing availability and security
Due to Salzgitter Group’s continuous manufacturing processes, it was difficult to keep on top of essential system maintenance, operating system upgrades and, most importantly, security patching.
Nicolas Otten, systems analyst and engineer at GESIS, elaborates: “Previously, we had to schedule downtime in order to update and patch individual IT systems. But because these systems support mission-critical workloads, and manufacturing operations run 24 hours a day, Salzgitter Group was always very reluctant to take them offline. Even a short period of downtime causes a huge amount of disruption, so convincing business leaders to let us take systems offline was always a challenge.
“To roll out a software update or apply a security patch, we used to have to take each system offline for 30 minutes. It may not sound like a long time, but any disruption to manufacturing operations is a real headache. When we did eventually get permission from Salzgitter Group to patch systems, it still took a lot of time and effort to schedule the downtime. We had to consult with lots of different parties, such as the SAP administration team, application managers and so on. There was a lot of back-and-forth. We estimate that it took about an hour in total just to arrange a maintenance window.”
Responding to emerging security threats
To meet Salzgitter Group’s expectation of round-the-clock service availability, GESIS only took systems offline when absolutely necessary for extended maintenance, usually about once a year.
Lowin confirms: “Because we were rarely able to take systems offline, we could not patch some systems very often. For the most important applications, we were only able to schedule one downtime a year, leaving some known issues unpatched for prolonged time periods. And with cybercrime on the rise in recent years, it’s more important than ever to update software and patch systems regularly for maximum protection against external threats.”
To ensure that it did not fall behind on systems security, GESIS wanted to be able to install patches more frequently. But how could the company do so while keeping downtime to a minimum?
“With SUSE Linux Enterprise Live Patching, we can complete security patching quickly during business hours. Because we can apply Linux kernel fixes without impacting the business-critical SAP applications running on the system, we don’t need to schedule downtime windows — we can patch systems practically whenever we want.”
SUSE Solution
Managing a large IT environment
GESIS turned to SUSE, one of its key technology partners, for advice. After discussing its objectives and technical requirements, the SUSE team introduced GESIS to SUSE Linux Enterprise (SLE) Live Patching — a cutting-edge technology that enables system administrators to apply Linux kernel fixes while systems are still running, without interrupting service.
Lowin comments: “We have relied on SUSE Linux Enterprise Server (SLES) to support IT operations for several years now. Our SLES landscape is remarkably stable, reliable and productive, helping us meet our business requirements. When we found out that SUSE had developed a complementary live patching solution, we were very keen to trial it.”
GESIS hosts all of Salzgitter Group’s mission-critical business systems, including SAP ERP applications and increasingly also SAP HANA databases, running on SLES for SAP Applications, together with general infrastructure services running on Microsoft Windows Server. The infrastructure is fully virtualized on 40 physical servers running VMware vSphere.
In total, GESIS runs approximately 500 virtual machines, of which around 225 host SAP software with SLES for SAP Applications and additional general business software. GESIS also runs about 25 dedicated physical servers with SLES for specific workloads that are not suitable for virtualization.
Understanding SUSE live patching
The SUSE live patching solution allows system administrators to install Linux kernel fixes on the fly with zero service interruption. It does this without the need to reboot systems, keeping business-critical applications running even in the event of critical kernel security updates.
SLE Live Patching is built on SUSE’s kGraft Linux kernel technology. SUSE developed kGraft with the Linux Community to improve protection of Linux systems without interruption. With in-memory databases such as SAP HANA becoming more and more widespread, the impact of restarts becomes even bigger, since these typically large workloads take longer to boot up fully. SLE Live Patching takes advantage of standard kernel features and combines them to patch systems during runtime. The solution leverages the built-in Ftrace debugging and analysis tool to monitor which kernel functions are being called. The SLE Live Patching technology then works by dynamically redirecting calls to kernel functions to a new, patched version of that function. This atomic approach ensures that no running software needs to be stopped, since the upgrade process is fully transparent to user-space processes (such as business applications) running outside of the Linux kernel.
Smooth and simple deployment
GESIS worked with a team from SUSE to understand and evaluate SLE Live Patching. Seamless integration with the company’s existing SUSE Manager solution made for a smooth deployment to all production systems.
Otten recalls: “We were pleasantly surprised by how quick and easy it was to roll out live kernel patching for SLES. We were able to get everything up and running in just a few hours. As usual, the support we received from SUSE was exceptional. They were on hand to answer any questions we had and gave us some top tips during the implementation. We were also very impressed with the level of detail in the solution documentation.”
Lowin adds: “The fact that you can easily integrate the Live Patching technology with SUSE Manager is a big bonus for us. This was a very fast process — it was as simple as adding a new update channel in SUSE Manager, and we could start using Live Patching right away.”
Thanks to SLE Live Patching, applications now keep running during updates because the patching is transparent from the perspective of the application running on the Linux system. This gives GESIS enormous flexibility, empowering system administrators to address serious vulnerabilities quickly and without compromising service availability. With live kernel patching for SLES, GESIS can deploy security and reliability patches to both physical and virtualized servers without having to take systems offline.
The Results
Improving system security
With SLE Live Patching, GESIS has almost eliminated the need for system downtime. Today, the company can roll out software updates and security patches as and when required, without any service disruption. This means that by not scheduling additional maintenance windows, GESIS can avoid potentially about 125 hours of accumulated downtime across all SAP applications for every crucial security patch.
Lowin remarks: “With SLE Live Patching, we can complete security patching quickly during business hours. Because we can apply Linux kernel fixes without impacting the business-critical SAP applications running on the system, we don’t need to schedule downtime windows — we can patch systems practically whenever we want. We have been using SLE Live Patching for a while now, and end users have never noticed any impact on their business applications. Having the flexibility to patch systems on the fly maximizes system uptime and service availability, ensuring that business operations tightly integrated with manufacturing processes run uninterrupted, while also saving us a great deal of coordination overhead and hassle.”
The possibility to patch faster also facilitates future certifications, for example, compliance with the ISO 27001 international information security standard.
Increasing efficiency and service availability
Thanks to SLE Live Patching, GESIS has reduced the number of times that it needs to reboot systems.
The company only schedules a single downtime window once a year in order to deal with major maintenance works and service pack updates.
Live kernel patching for SLES has enabled GESIS and Salzgitter Group to cut coordination costs between IT and business departments for every Linux kernel patch deployment, saving the teams 50% in communication and administration overhead.
Otten says: “Not having to take core systems down for 20,000 users to apply small, but nonetheless very important, security patches and fixes is a huge weight off our shoulders. That way, we avoid the stress and inconvenience of scheduling unwanted downtime with the business. Also, we and our colleagues no longer need to work weekend shifts to install patches — a major benefit for our work-life balance.
“Crucially, we can patch systems much more frequently than we could previously, helping to ensure that systems are secure. Now, we typically patch all systems every six to eight weeks. It is also reassuring to know that we are able to quickly patch a security vulnerability immediately when it emerges. So, rather than putting off Linux kernel patches until the next planned downtime window, as we often would in the past, we can now apply them right away without any impact on operations. This minimizes risk exposure and enhances security, helping us to keep systems protected against internal and external threats.”
Alongside SLE Live Patching, using SUSE Manager has considerably reduced manual workload for GESIS.
Lowin explains: “Before we started using SUSE Manager, we had to patch every single server individually. With SUSE Manager, we have a user-friendly interface that tells us which of our servers need patches most urgently, making our lives much easier and server updates 75% faster. Thanks to the tight integration with SLE Live Patching, we can also track, schedule and roll out security patches directly via SUSE Manager. Keeping systems protected couldn’t be any simpler.”
Establishing a new standard
SLE Live Patching has quickly become the standard configuration option at GESIS. The company must ensure the highest possible level of availability for Salzgitter Group’s core business applications; live kernel patching for Linux Enterprise Server helps GESIS to achieve this goal.
Otten says: “Salzgitter Group relies on us to keep all IT systems running smoothly around the clock for its global operations. Thanks to SLE Live Patching, we no longer have to take business-critical SAP business applications offline to apply patches. These applications are not just used to support finance and back-office administration, but also manage HR self-service functions and are connected with 24/7 steel manufacturing operations at sites all-around the world. SLE Live Patching enables us to avoid delays in basic processes like holiday booking for 20,000 employees while also keeping integrated production lines running reliably.”
Lowin concludes: “We have been using SLE Live Patching for a couple of years now, and we are highly satisfied. It’s an extremely valuable solution for GESIS, as it enables us to improve security and service availability without additional management or administration overhead — helping us to keep Salzgitter Group’s production lines rolling.”