It has many names – hot fixing, live patching, runtime patching, rebootless updates, concurrent updates. It’s a holy grail of uptime.
It is longed for by scientists who really do not want to stop a simulation that has been running for the past few months – just because of a needed kernel stability fix. IT staff who run their machines without critical security patches, because the departments they serve cannot agree on a good time for scheduled downtime, dream about it in their sleepless nights.
And it is a technology that is not available in the upstream Linux kernel, and by extension none of the major Linux distributions provide updates this way.
Yes, there are at least two technologies that can achieve the goal: Ksplice and OpenVZ Checkpointing. The first will probably never make it upstream, among other reasons because the open source version is no longer updated. The second needs quite some infrastructure and will never be able to apply the patch without a short but visible interruption of service.
Enter kGraft. Developed by SUSE Labs (*) as a research project, currently just an unpolished prototype, kGraft is surprisingly simple. But working well.
kGraft builds on technologies and ideas that are already present in the kernel: ftrace and its mcount-based reserved space in function headers, the INT3/IPI-NMI patching also used in jumplabels, and RCU-like update of code that does not require stopping the kernel. A kGraft patch is a kernel module and fully relies on the in-kernel module loader to link the new code with the kernel. Thanks to all that, the design can be nicely minimalistic.
While kGraft is, by choice, limited to replacing whole functions and constants they reference, this does not limit the set of code patches that can be applied significantly. kGraft will offer tools to assist in creating the live patch modules, identifying which functions need to be replaced based on a patch, and creating the patch module source code.
We plan a first kGraft release in March, under GPLv3 for parts that touch GCC and GPLv2 for Linux kernel parts, and we aim at getting it merged fully into the upstream projects.
We’ll be going deeper into how it works in further postings on this blog and also at the Collaboration Summit in Napa Valley in March.
(*) Vojtěch Pavlík, Jiří Kosina, Jiří Slabý, Petr Mládek