kGraft: Live Kernel Patching | SUSE Communities

kGraft: Live Kernel Patching


Editor’s Note: kGraft is now known as Live Patching.

It has many names – hot fixing, live patching, runtime patching, rebootless updates, concurrent updates.  It’s a holy grail of uptime.

It is longed for by scientists who really do not want to stop a simulation that has been running for the past few months – just because of a needed kernel stability fix.  IT staff who run their machines without critical security patches, because the departments they serve cannot agree on a good time for scheduled downtime, dream about it in their sleepless nights.

And it is a technology that is not available in the upstream Linux kernel, and by extension none of the major Linux distributions provide updates this way.

Yes, there are at least two technologies that can achieve the goal: Ksplice and OpenVZ Checkpointing.  The first will probably never make it upstream, among other reasons because the open source version is no longer updated.  The second needs quite some infrastructure and will never be able to apply the patch without a short but visible interruption of service.

Enter kGraft. Developed by SUSE Labs (*) as a research project, currently just an unpolished prototype, kGraft is surprisingly simple.  But working well.

kGraft builds on technologies and ideas that are already present in the kernel: ftrace and its mcount-based reserved space in function headers, the INT3/IPI-NMI patching also used in jumplabels, and RCU-like update of code that does not require stopping the kernel.  A kGraft patch is a kernel module and fully relies on the in-kernel module loader to link the new code with the kernel.  Thanks to all that, the design can be nicely minimalistic.

While kGraft is, by choice, limited to replacing whole functions and constants they reference, this does not limit the set of code patches that can be applied significantly.  kGraft will offer tools to assist in creating the live patch modules, identifying which functions need to be replaced based on a patch, and creating the patch module source code.

We plan a first kGraft release in March, under GPLv3 for parts that touch GCC and GPLv2 for Linux kernel parts, and we aim at getting it merged fully into the upstream projects.

We’ll be going deeper into how it works in further postings on this blog and also at the Collaboration Summit in Napa Valley in March.

(*) Vojtěch Pavlík, Jiří Kosina, Jiří Slabý, Petr Mládek

(Visited 8 times, 1 visits today)


  • Avatar photo Loic Devulder says:

    Is kGraft will be available in SLES12?

    • Avatar photo Vojtěch Pavlík says:

      kGraft is obviously a technology most useful in Enterprise deployments, and SUSE has customers asking for it.

      Yet, let’s not do the second step before the first.

      Discussing the approach on technical merits within the broader community, gathering feedback and criticism, as well as improving the solution based on it to be able to gain upstream acceptance is a mandatory step for us.

      Only after that we can include the technology in a product offering.

  • Avatar photo matwey says:

    What will happen if I have some struct definition and a set of functions receiving pointer to the structure. Then I have to introduce additional field into the struct and modify the functions in order to fix an issue.
    Then I use live patching and… I have objects allocated in kernel as old version of the struct and new code receiving pointers to the objects allocated as new version.
    How are you going to perform migration of data? In the situation described it is required to reallocate all objects of a given structure type.

    • Avatar photo Vojtěch Pavlík says:

      Transforming data from pre-patch format to a post-patch format is in the area of semantic change and so far, no generic automated solution exists.

      In case the data is managed in some kind of a structure (linked lists, trees, etc.), one can use the locking mechanism of the structure to obtain exclusive access and perform the transformation.

      In case the data isn’t centrally managed (eg. spin locks), then the only way is to instrument every function accessing said data and keep versions inside the data, converting on first access. In this case, it isn’t possible to tell whether all the data was converted and thus the instrumentation can never be removed, thus having a performance impact.

      kGraft doesn’t by itself use any of the above approaches, but lets a developer use them manually, requiring new code to be developed for a specific patch, should there be a need.

      Fortunately, most security patches don’t require such in memory data conversions.

      • Avatar photo Matwey Kornilov says:

        Thank you for the reply. But what if the function to be patched is used as callback. Will we be able to replace the pointer to the function which is stored in different structures around the kernel?

        • Avatar photo Vojtěch Pavlík says:

          The trick there is that it’s not the callers that are patched to call a new version of the function, it’s the very first few bytes of the old function that get replaced by a long jump instruction. It means an extra jump, but it also means any function pointer, even those embedded in structures, remains valid and will, through the jump, call the new version of the function.

  • Avatar photo ntpl1 says:

    Ksplice is very much available for enterprise environments today. Last week, my team downloaded Oracle Linux and Ksplice is a complimentary tool that is available with the support program Oracle offers for Linux support; It’s a fully functional tool available with the mainline kernel and I’m really impressed with the way it functions!!

    • Avatar photo Vojtěch Pavlík says:

      Dear Nawaz Pasha, please give my greetings also to the rest of the Oracle sales team!

      SUSE is committed to be a team player in the open source world, and to guaranteeing choice in the Enterprise Linux ecosystem. We strive to get all our kernel technologies accepted in the upstream Linux kernel.

      That is why we chose to develop kGraft.

  • Avatar photo estevaovaladao says:

    Awesome solution. Thanks.

  • Avatar photo vad says:

    Actually, kernelcare released production ready product for online security patches like that.

    ksplice is available to Oracle Linux users only.. 🙁

    After looking into kgraft sources – I have a question why do you believe RCU-like applying patches is safe at all?… It doesn’t look so!

    • Avatar photo Vojtěch Pavlík says:

      Sadly, CloudLinux’s KernelCare solution is closed-source and thus not viable for upstream acceptance.

      Regarding the RCU-like method of applying patches: it is safe in regards to changes in semantics or format of function calls. It doesn’t provide safety against semantic or format changes of in-memory data, but neither does the stop-and-check-stacks approach: both of the approaches need special handling of those cases.

      Do you have any specific concerns regarding the RCU-like method?

  • Avatar photo paigethylamine says:

    I’d love to use it except I already made up my mind to use grsec and I doubt the two will work together.

    • Avatar photo Vojtěch Pavlík says:

      There is nothing that would prevent using kGraft with grsec, of course unless you disable loading of kernel modules.

  • Avatar photo seife says:

    “kGraft will offer tools to assist in creating the live patch modules, identifying which functions need to be replaced based on a patch, and creating the patch module source code.”

    I have trouble finding these tools anywhere. Care to share a link to github or such?

    Best regards

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar photo