Live Kernel Patching: Modern Tools


Back in 2014, the best (if not only) option for patching the Linux kernel without rebooting was KernelCare, a tool developed by our partners at Cloud Linux.

The situation has since changed quite a bit as live patching has officially been included in the kernel as of version 4.0. The tools kpatch and kGraft, which were still in development in 2014, have also been massively improved. Kpatch was even added to the official repository and in Ubutnu 16.04, it can be installed from the default package manager. Canonical has also recently released their Canonical Livepatch Service, which can be used to patch the Ubuntu kernel without rebooting.

In today’s article, we’ll be looking at some of these modern patching tools and how they can be used in practice.


We’ll start with a simple experiment. For this, we’ll need a Linux distribution with a kernel that’s ver. 4.0 or higher (we’ll be using Ubuntu 16.04 throughout this article for our examples). In newer kernel versions, livepatch is included by default.

To see how it works, we’ll first have to install kernel headers:

$ sudo apt-get install linux-headers-$(uname -r)

Then we install kernel debug symbols:

#we add the repository
$ codename=$(lsb_release -sc)
$ sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb ${codename} main restricted universe multiverse
deb ${codename}-security main restricted universe multiverse
deb ${codename}-updates main restricted universe multiverse
deb ${codename}-proposed main restricted universe multiverse
#add our key
wget -Nq -O- | sudo apt-key add -
#update the package list
$ sudo apt-get update
#and install debug symbols
$ sudo apt-get install linux-image-$(uname -r)-dbgsym

Next we execute:

$ sudo sed -i -- 's/#deb-src/deb-src/g' /etc/apt/sources.list && sudo sed -i -- 's/# deb-src/deb-src/g' /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get build-dep linux

Now everything is ready and we can start our experiment. We start by running the following command:


This downloads the source code for a kernel module that changes our kernel’s code and modifies the cat /proc/cmdline output. Now we need to build the module. To do this, we create a makefile:

obj-m +=livepatch-sample.o
KDIR= /lib/modules/$(shell uname -r)/build
        $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
        rm -rf *.o *.ko *.mod.* .c* .t*

Then, we compile the module and insert it into the kernel.

$ make
$ insmod livepatch-sample.ko

Let’s see what we’ve got. We run:

$ cat /proc/cmdinfo

Instead of the standard kernel parameters, we get the following text:

this has been live patched

As we can see, our kernel has been successfully patched.

All configurations about downloaded patches are saved in the directory /sys/kernel/livepatch:

$ ls /sys/kernel/livepatch/

We can deactivate the patch using the command:

$ echo 0 > /sys/kernel_livepatch/livepatch_sample/enabled


Kpatch was developed by Red Hat and got its first wide release in February 2016. Since then, it has seen major improvements and and has already been included in the Ubuntu 16.04 official repository. We’ll look at how kpatch is used with some practical examples.

We’ll start by installing the necessary dependencies:

$ sudo apt-get install libelf-dev dpkg-dev 

To get the most out of kpatch, we should also install ccache:

$ sudo apt-get install ccache
$ ccache --max-size=5G

Now that we have our dependencies, we can install kpatch:

$ sudo apt-get install kpatch kpatch-build

In our experiment, we’ll be patching the kernel sources. We clone the repository with our version of of the Ubuntu kernel:

git clone git://

Afterwards, we copy the source code to the directory ubuntu-xenial-kpatch (this so we can modify the source code and then make patches based on the changes):

$ mkdir ubuntu-xenial-kpatch
$ cp -r ubuntu-xenial ubuntu-xenial-kpatch

We open the file ubuntu-xenial-kpatch/ubuntu-xenial/fs/proc/version.c and make the following changes:

static int version_proc_show(struct seq_file *m, void *v)
        seq_printf(m, linux_proc_banner,
                "This has been patched!",
        return 0;

We create our patch by running the command:

$ diff -u ubuntu-xenial/fs/proc/version.c  ubuntu-xenial.kpatch/ubuntu-xenial/proc.version.c > version.patch

The patch is an ordinary text file that lists the modifications:

--- ubuntu-xenial/fs/proc/version.c     2016-12-05 10:04:30.126141156 +0300
+++ ubuntu-xenial.kpatch/ubuntu-xenial/fs/proc/version.c        2016-12-05 10:10:35.678461801 +0300
@@ -8,6 +8,7 @@
 static int version_proc_show(struct seq_file *m, void *v)
        seq_printf(m, linux_proc_banner,
+               "This has been patched!",

To insert our patch into the kernel, we run:

$ kpatch-build -t vmlinux --skip-gcc-check version.patch
WARNING: Skipping gcc version matching check (not recommended)
Debian/Ubuntu distribution detected
Downloading the kernel source for 4.4.0-51-generic
Unpacking kernel source
Testing patch file
checking file fs/proc/version.c
Reading special section data
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
version.o: changed function: version_proc_show
Building patch module: kpatch-version.ko

From the printout we can see that we’ve successfully created a kernel module. We can apply it using the standard method:

sudo insmod kpatch-version.ko

Let’s take a look at the results:

cat /proc/version
This has been patched! version Linux (buildd@lcy01-08) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) 4.4.0-51-generic

That’s it! We have successfully patched our kernel.

Canonical Livepatch Service

A few months ago, the company Canonical launched their official Canonical LivePatch Service, which lets you patch the kernel on the fly with simple commands. As the service is mainly oriented towards enterprise-class users, it is a paid service.

Your everyday users can also get the latest kernel updates. For this, you need to register on Ubuntu One and get a token. Tokens let you install canonical-livepatch, which downloads and applies patches, on 3 machines.

Let’s look at how Canonical Livepatch Service works. We’ll follow the link above, get our token, and then run:

$ sudo snap install canonical-livepatch

Once the installation is complete, we exit the system, log back in and run:

$ sudo canonical-livepatch enable [token]

If we’ve done everything right, we’ll see the following message:

Successfully enabled device. Using machine-token: [token]

Then we run:

$ canonical-livepatch status
kernel: 4.4.0-47.68-generic
fully-patched: true
version: "14.1"

The output tells us that canonical-livepatch is working and the kernel is up to date. We can get more detailed information using the option –verbose:

$ canonical-livepatch status --verbose
client-version: "6"
machine-id: [id]
architecture: x86_64
cpu-model: Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
last-check: 2016-12-05T11:56:02.88803394+03:00
boot-time: 2016-11-29T10:48:38+03:00
uptime: 145h18m21s
- kernel: 4.4.0-47.68-generic
  running: true
    checkState: checked
    patchState: applied
    version: "14.1"
    fixes: |-
      * CVE-2016-7425
      * CVE-2016-8658

We can also view information on installed patches by looking in the directory /sys/kernel/livepatch:

$ ls  /sys/kernel/livepatch

Kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14 is the latest downloaded patch. The last numbers in the patch name (14) match the version number given in the printout from canonical-livepatch status (see above).

We can ensure the new patch was added using the command lsmod:

$ lsmod |grep livepatch
kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14    36864  1


In this article, we looked at a few tools for patching the Linux kernel on the fly. Naturally, we couldn’t cover every aspect of these tools in one article. If you have anything you want to add, please feel free to do so in the comments below.

And if you’d like to learn more, please look at the following links:

Leave a Reply

Your email address will not be published. Required fields are marked *