Main    /    Blog    /    Live Kernel Patching: Modern Tools

←   →

Live Kernel Patching: Modern Tools

18 January 2017 7 min read
Tags: , , , , ,

pr-3322

Back in 2014, the best (if not only) option for patching the Linux kernel without rebooting was KernelCare, a tool developed by our partners at Cloud Linux.

The situation has since changed quite a bit as live patching has officially been included in the kernel as of version 4.0. The tools kpatch and kGraft, which were still in development in 2014, have also been massively improved. Kpatch was even added to the official repository and in Ubutnu 16.04, it can be installed from the default package manager. Canonical has also recently released their Canonical Livepatch Service, which can be used to patch the Ubuntu kernel without rebooting.

In today’s article, we’ll be looking at some of these modern patching tools and how they can be used in practice.

livepatch

We’ll start with a simple experiment. For this, we’ll need a Linux distribution with a kernel that’s ver. 4.0 or higher (we’ll be using Ubuntu 16.04 throughout this article for our examples). In newer kernel versions, livepatch is included by default.

To see how it works, we’ll first have to install kernel headers:

$ sudo apt-get install linux-headers-$(uname -r)

Then we install kernel debug symbols:

#we add the repository
$ codename=$(lsb_release -sc)
$ sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF
 
#add our key
wget -Nq http://ddebs.ubuntu.com/dbgsym-release-key.asc -O- | sudo apt-key add -
 
#update the package list
$ sudo apt-get update
 
#and install debug symbols
$ sudo apt-get install linux-image-$(uname -r)-dbgsym

Next we execute:

$ sudo sed -i -- 's/#deb-src/deb-src/g' /etc/apt/sources.list && sudo sed -i -- 's/# deb-src/deb-src/g' /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get build-dep linux

Now everything is ready and we can start our experiment. We start by running the following command:

wget http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/samples/livepatch/livepatch-sample.c

This downloads the source code for a kernel module that changes our kernel’s code and modifies the cat /proc/cmdline output. Now we need to build the module. To do this, we create a makefile:

obj-m +=livepatch-sample.o
KDIR= /lib/modules/$(shell uname -r)/build
all:
        $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
        rm -rf *.o *.ko *.mod.* .c* .t*

Then, we compile the module and insert it into the kernel.

$ make
$ insmod livepatch-sample.ko

Let’s see what we’ve got. We run:

$ cat /proc/cmdinfo

Instead of the standard kernel parameters, we get the following text:

this has been live patched

As we can see, our kernel has been successfully patched.

All configurations about downloaded patches are saved in the directory /sys/kernel/livepatch:

$ ls /sys/kernel/livepatch/
livepatch_sample

We can deactivate the patch using the command:

$ echo 0 > /sys/kernel_livepatch/livepatch_sample/enabled

Kpatch

Kpatch was developed by Red Hat and got its first wide release in February 2016. Since then, it has seen major improvements and and has already been included in the Ubuntu 16.04 official repository. We’ll look at how kpatch is used with some practical examples.

We’ll start by installing the necessary dependencies:

$ sudo apt-get install libelf-dev dpkg-dev

To get the most out of kpatch, we should also install ccache:

$ sudo apt-get install ccache
$ ccache --max-size=5G

Now that we have our dependencies, we can install kpatch:

$ sudo apt-get install kpatch kpatch-build

In our experiment, we’ll be patching the kernel sources. We clone the repository with our version of of the Ubuntu kernel:

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git

Afterwards, we copy the source code to the directory ubuntu-xenial-kpatch (this so we can modify the source code and then make patches based on the changes):

$ mkdir ubuntu-xenial-kpatch
$ cp -r ubuntu-xenial ubuntu-xenial-kpatch

We open the file ubuntu-xenial-kpatch/ubuntu-xenial/fs/proc/version.c and make the following changes:

#include  	
#include  	
#include  	
#include  	
#include  	
#include  	
 
static int version_proc_show(struct seq_file *m, void *v)
{
        seq_printf(m, linux_proc_banner,
                "This has been patched!",
                utsname()->sysname,
                utsname()->release,
                utsname()->version);
        return 0;
}

We create our patch by running the command:

$ diff -u ubuntu-xenial/fs/proc/version.c  ubuntu-xenial.kpatch/ubuntu-xenial/proc.version.c > version.patch

The patch is an ordinary text file that lists the modifications:

--- ubuntu-xenial/fs/proc/version.c     2016-12-05 10:04:30.126141156 +0300
+++ ubuntu-xenial.kpatch/ubuntu-xenial/fs/proc/version.c        2016-12-05 10:10:35.678461801 +0300
@@ -8,6 +8,7 @@
 static int version_proc_show(struct seq_file *m, void *v)
 {
        seq_printf(m, linux_proc_banner,
+               "This has been patched!",
                utsname()->sysname,
                utsname()->release,
                utsname()->version);

To insert our patch into the kernel, we run:

$ kpatch-build -t vmlinux --skip-gcc-check version.patch
 
WARNING: Skipping gcc version matching check (not recommended)
Debian/Ubuntu distribution detected
Downloading the kernel source for 4.4.0-51-generic
Unpacking kernel source
Testing patch file
checking file fs/proc/version.c
Reading special section data
Building original kernel
Building patched kernel
Detecting changed objects
Rebuilding changed objects
Extracting new and modified ELF sections
version.o: changed function: version_proc_show
Building patch module: kpatch-version.ko
SUCCESS

From the printout we can see that we’ve successfully created a kernel module. We can apply it using the standard method:

sudo insmod kpatch-version.ko

Let’s take a look at the results:

cat /proc/version
This has been patched! version Linux (buildd@lcy01-08) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) 4.4.0-51-generic

That’s it! We have successfully patched our kernel.

Canonical Livepatch Service

A few months ago, the company Canonical launched their official Canonical LivePatch Service, which lets you patch the kernel on the fly with simple commands. As the service is mainly oriented towards enterprise-class users, it is a paid service.

Your everyday users can also get the latest kernel updates. For this, you need to register on Ubuntu One and get a token. Tokens let you install canonical-livepatch, which downloads and applies patches, on 3 machines.

Let’s look at how Canonical Livepatch Service works. We’ll follow the link above, get our token, and then run:

$ sudo snap install canonical-livepatch

Once the installation is complete, we exit the system, log back in and run:

$ sudo canonical-livepatch enable [token]

If we’ve done everything right, we’ll see the following message:

Successfully enabled device. Using machine-token: [token]

Then we run:

$ canonical-livepatch status
 
kernel: 4.4.0-47.68-generic
fully-patched: true
version: "14.1"

The output tells us that canonical-livepatch is working and the kernel is up to date. We can get more detailed information using the option –verbose:

$ canonical-livepatch status --verbose
 
 
client-version: "6"
machine-id: [id]
machine-token:[token]
architecture: x86_64
cpu-model: Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
last-check: 2016-12-05T11:56:02.88803394+03:00
boot-time: 2016-11-29T10:48:38+03:00
uptime: 145h18m21s
status:
- kernel: 4.4.0-47.68-generic
  running: true
  livepatch:
    checkState: checked
    patchState: applied
    version: "14.1"
    fixes: |-
      * CVE-2016-7425
      * CVE-2016-8658

We can also view information on installed patches by looking in the directory /sys/kernel/livepatch:

$ ls  /sys/kernel/livepatch
kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14

Kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14 is the latest downloaded patch. The last numbers in the patch name (14) match the version number given in the printout from canonical-livepatch status (see above).

We can ensure the new patch was added using the command lsmod:

$ lsmod |grep livepatch
kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14    36864  1

Conclusion

In this article, we looked at a few tools for patching the Linux kernel on the fly. Naturally, we couldn’t cover every aspect of these tools in one article. If you have anything you want to add, please feel free to do so in the comments below.

And if you’d like to learn more, please look at the following links:

18 January 2017 Tags: , , , , ,

Popular articles