Live Patching Case Study of GESIS

You can save time and resources by using Live Patching. GESIS is one of the many organizations who achieved excellent results using SUSE Linux Enterprise Live Patching. Here we outline some of those results so you can make an assessment about how these can apply to your environment.

First, let’s start with “What is Live Patching”?

SUSE Linux Enterprise Live Patching is a cutting-edge technology that enables system administrators to apply Linux kernel fixes while systems are still running, without interrupting service. To get a better understanding of how a system gets patched, checkout the recent blogs: Live Patching Meltdown – SUSE Engineer’s research project (Part 1) and Business-critical systems demand Live Patching capability .

What benefits should you typically expect from Live Patching?

First, from an IT administrator’s point of view. 

The IT admins will see a reduction in downtime.

• By eliminating unplanned downtimes – No need to plan for a downtime to apply patches. You can apply security patches to the kernel as soon as they become available.
• Reducing planned downtimes – No need to reboot system when applying kernel patches. You will save the time required to schedule and perform the planned downtime.
• Saving hours of reboot downtime – For database applications such as SAP HANA, you can avoid the long time needed to reboot the systems. A typical SAP HANA system has TBs of memory that need to be moved to storage, so a cluster of such systems could take several hours to reboot. For large virtualization environments, it can take a lot of admin time in restarting virtual machine environments and bringing all running applications in the VMs to the same point as before.

 Overall, organizations typically save several man months of administration time.

Business perspective.

You can keep your business up and running.

• By maintaining continuity of services at all times – You will increase service availability by applying critical kernel patches with zero interruption of your business-critical applications.

Security point of view.

CISO will see an enhancement in security and compliance.

• By reduced exposure to security vulnerabilities by applying security patches as soon as possible.
• Audit applied patches using Live Patching’s unique ability to review the source code when applying patches.
• Stay compliant by keeping systems always up-to-date with patches.

Who is GESIS and what does their business-critical environment look like?

GESIS is part of Salzgitter Group, one of the largest steel producers in Europe with approximately 25,000 employees and annual sales of EUR 9 billion. The group consists of more than 150 subsidiaries worldwide, and GESIS is the central IT services provider to all group companies.

GESIS IT operations run 24 hours a day, 365 days a year. To keep production lines rolling and finished products on the move to customers without delays, the company depends on IT systems. In this highly automated and closely connected manufacturing and logistics environment, mission-critical SAP business applications are particularly important to ensure the business runs reliably – day in, day out.

The problems faced by GESIS are typical of many IT organizations

1. Downtime costs a lot!

To support 24/7 production, Salzgitter Group depends on the constant availability and robust service of its IT systems. Every minute of unplanned downtime would cost thousands of Euros and jeopardise complex production schedules of diverse steel products.

2. It is not easy to do security patching on live systems.

Due to Salzgitter Group’s continuous manufacturing processes, it was difficult to keep on top of essential system maintenance, operating system upgrades and, most importantly, security patching.

3. Scheduling downtime is difficult AND requires going thru approval process.

GESIS IT teams had to schedule downtime in order to update and patch individual IT systems. But because these systems support mission-critical workloads, and manufacturing operations run 24 hours a day, Salzgitter Group was always very reluctant to take them offline. Even a short period of downtime causes a huge amount of disruption, so convincing business leaders to let IT team take systems offline was always a challenge.

“To roll out a software update or apply a security patch, we used to have to take each system offline for 30 minutes. It may not sound like a long time, but any disruption to manufacturing operations is a real headache. When we did eventually get permission from Salzgitter Group to patch systems, it still took a lot of time and effort to schedule the downtime. We had to consult with lots of different parties, such as the SAP administration team, application managers and so on. There was a lot of back-and-forth. We estimate that it took about an hour in total just to arrange a maintenance window.” Thomas Lowin, Server Operations Lead, GESIS

4. It is a hard balancing act between security and productivity.

If you keep system running with security holes and don’t patch, security exposure grows quickly.

If you bring systems down to patch often then you lose productivity.

“Because we were rarely able to take systems offline, we could not patch some systems very often. For the most important applications, we were only able to schedule one downtime a year, leaving some known issues unpatched for prolonged time periods.” Thomas Lowin, Server Operations Lead, GESIS

What are some of the tangible benefits derived by GESIS that can also apply to your organization.

1. No need to schedule downtime for patches. Save time to plan & schedule a downtime and then execute the downtime.

“With SUSE Linux Enterprise Live Patching, we can complete security patching quickly during business hours. Because we can apply Linux kernel fixes without impacting the business-critical SAP applications running on the system, we don’t need to schedule downtime windows – we can patch systems practically whenever we want.” Thomas Lowin, Server Operations Lead, GESIS

2. Avoided potentially 125 hours of accumulated downtime for every security patch.
3. Saved 50% in communication and administration overhead by cutting coordination costs between IT and business departments.
4. Enhancement in security by ensuring that systems stay up to date with the latest Linux kernel patches.
5. Maximized service availability by eliminating the need to take systems offline for Linux kernel patching.

When should you start using Live Patching?

Live Patching is one of the technology jewels of SUSE Linux Enterprise product family. It is also one of the easiest to deploy and derive benefits.

If you are new to SUSE Linux Enterprise Server, you may want to give it a try as soon as you deploy SLE Server.

If you have been using SLE Server for years, you are missing out on a key technology innovation. You may want to try Live Patching earlier than later. To put it very simply and directly, once you know what Live Patching is and what it can do for your IT environment then we hope you will have very good reasons for NOT using Live Patching.

Resources for quick reference

• Detailed Case Study – GESIS
• Product: SUSE Linux Enterprise Live Patching
• Blog series detailing the journey of SUSE engineering team in live patching the famous security issue of “Meltdown”. These 4 blogs take you through the in-depth technical details of how the patching was done and lessons learnt. Certainly, a must read on Live Patching.

Live Patching Meltdown – SUSE Engineer’s research project (Part 1 – Key technical obstacles for Live Patching Meltdown)

Part 2 – Virtual address mappings and the Meltdown vulnerability

Part 3 – Changes needed for Translation Lookaside Buffer (TLB) flushing primitives

Part 4 – The Conclusion

• (Video) Live Patching Explained in 2 Mins.
• Other blogs
  • Business-critical systems demand Live Patching capability
  • Live Patching helps deliver nonstop IT with IBM Power Systems
  • Live Patching superhero doesn’t reboot from Stack Clash

Keep in touch @RajMeel7