SUSE Conversations


kGraft: Live Kernel Patching



By: Vojtěch Pavlík

January 31, 2014 7:49 am

Reads:18783

Comments:9

Rating:5.0

Editor’s Note: kGraft is now available for download.

It has many names – hot fixing, live patching, runtime patching, rebootless updates, concurrent updates.  It’s a holy grail of uptime.

It is longed for by scientists who really do not want to stop a simulation that has been running for the past few months – just because of a needed kernel stability fix.  IT staff who run their machines without critical security patches, because the departments they serve cannot agree on a good time for scheduled downtime, dream about it in their sleepless nights.

And it is a technology that is not available in the upstream Linux kernel, and by extension none of the major Linux distributions provide updates this way.

Yes, there are at least two technologies that can achieve the goal: Ksplice and OpenVZ Checkpointing.  The first will probably never make it upstream, among other reasons because the open source version is no longer updated.  The second needs quite some infrastructure and will never be able to apply the patch without a short but visible interruption of service.

Enter kGraft. Developed by SUSE Labs (*) as a research project, currently just an unpolished prototype, kGraft is surprisingly simple.  But working well.

kGraft builds on technologies and ideas that are already present in the kernel: ftrace and its mcount-based reserved space in function headers, the INT3/IPI-NMI patching also used in jumplabels, and RCU-like update of code that does not require stopping the kernel.  A kGraft patch is a kernel module and fully relies on the in-kernel module loader to link the new code with the kernel.  Thanks to all that, the design can be nicely minimalistic.

While kGraft is, by choice, limited to replacing whole functions and constants they reference, this does not limit the set of code patches that can be applied significantly.  kGraft will offer tools to assist in creating the live patch modules, identifying which functions need to be replaced based on a patch, and creating the patch module source code.

We plan a first kGraft release in March, under GPLv3 for parts that touch GCC and GPLv2 for Linux kernel parts, and we aim at getting it merged fully into the upstream projects.

We’ll be going deeper into how it works in further postings on this blog and also at the Collaboration Summit in Napa Valley in March.

(*) Vojtěch Pavlík, Jiří Kosina, Jiří Slabý, Petr Mládek

VN:D [1.9.22_1171]
Rating: 5.0/5 (13 votes cast)
kGraft: Live Kernel Patching, 5.0 out of 5 based on 13 ratings

Tags: , , , , , , , ,
Categories: Announcements, Enterprise Linux, Expert Views

Disclaimer: As with everything else at SUSE Conversations, this content is definitely not supported by SUSE (so don't even think of calling Support if you try something and it blows up).  It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.

9 Comments

  1. By:Loic Devulder

    Is kGraft will be available in SLES12?

    • By:Vojtěch Pavlík

      kGraft is obviously a technology most useful in Enterprise deployments, and SUSE has customers asking for it.

      Yet, let’s not do the second step before the first.

      Discussing the approach on technical merits within the broader community, gathering feedback and criticism, as well as improving the solution based on it to be able to gain upstream acceptance is a mandatory step for us.

      Only after that we can include the technology in a product offering.

  2. By:matwey

    What will happen if I have some struct definition and a set of functions receiving pointer to the structure. Then I have to introduce additional field into the struct and modify the functions in order to fix an issue.
    Then I use live patching and… I have objects allocated in kernel as old version of the struct and new code receiving pointers to the objects allocated as new version.
    How are you going to perform migration of data? In the situation described it is required to reallocate all objects of a given structure type.

    • By:Vojtěch Pavlík

      Transforming data from pre-patch format to a post-patch format is in the area of semantic change and so far, no generic automated solution exists.

      In case the data is managed in some kind of a structure (linked lists, trees, etc.), one can use the locking mechanism of the structure to obtain exclusive access and perform the transformation.

      In case the data isn’t centrally managed (eg. spin locks), then the only way is to instrument every function accessing said data and keep versions inside the data, converting on first access. In this case, it isn’t possible to tell whether all the data was converted and thus the instrumentation can never be removed, thus having a performance impact.

      kGraft doesn’t by itself use any of the above approaches, but lets a developer use them manually, requiring new code to be developed for a specific patch, should there be a need.

      Fortunately, most security patches don’t require such in memory data conversions.

      • By:Matwey Kornilov

        Thank you for the reply. But what if the function to be patched is used as callback. Will we be able to replace the pointer to the function which is stored in different structures around the kernel?

        • By:Vojtěch Pavlík

          The trick there is that it’s not the callers that are patched to call a new version of the function, it’s the very first few bytes of the old function that get replaced by a long jump instruction. It means an extra jump, but it also means any function pointer, even those embedded in structures, remains valid and will, through the jump, call the new version of the function.

  3. By:ntpl1

    Ksplice is very much available for enterprise environments today. Last week, my team downloaded Oracle Linux and Ksplice is a complimentary tool that is available with the support program Oracle offers for Linux support; It’s a fully functional tool available with the mainline kernel and I’m really impressed with the way it functions!!

    • By:Vojtěch Pavlík

      Dear Nawaz Pasha, please give my greetings also to the rest of the Oracle sales team!

      SUSE is committed to be a team player in the open source world, and to guaranteeing choice in the Enterprise Linux ecosystem. We strive to get all our kernel technologies accepted in the upstream Linux kernel.

      That is why we chose to develop kGraft.

  4. By:estevaovaladao

    Awesome solution. Thanks.

Comment

RSS