Low Latency Kernel

TL;DR
Give applications permission to not play nice with others: /sbin/setcap capipclock,capsysnice=ep /path/to/greedy/application

A computer's CPU does a lot, cycling constantly through series of loops and commands and probes and checks and refreshes and updates so consistently that sys admin even today talk in terms of “CPU cycles” when estimating the “cost” of a computationally intensive project. Regardless of what you do artistically on your computer, yours is no exception: the CPU is in high demand every second, and not everything demanding its attention can be tended to at the same time. For this reason, the driver of the CPU (the “kernel”) queues all tasks needing attention, delegates priorities to each one, and then executes chunks of each waiting task according to priority. This happens millions times per second.

For most everyday computing, the Linux kernel's somewhat conservative policy of what is important and what can wait another millisecond or two before getting worked on is perfect; onscreen effects get displayed smoothly, music plays seamlessly, the desktop clock keeps ticking, and no one is ever aware that user tasks are secretly being preempted for boring low-level computer tasks.

One place that this does not do well is professional audio, or more specifically, professional audio with realtime effects or realtime recording. Most other artistic tasks (even video effects) are perfectly acceptable on default settings, but pro audio requires high priority in order to playback smoothly without clicks, stutters, or breaks (called “xruns”). Technically speaking, this means that audio must gain the permission to preempt the usual priorities of running tasks as needed. Normally, this is not allowed on a Linux system, but as of kernel version 2.6.38.4 (so any 3.x or higher kernel) the ability to whitelist preemption was written into the OS.

As long as you are running a Linux kernel versioned 3 or higher, you have a preemptive kernel already and there is no need to look for a special “low latency” kernel. You are, however, required to whitelist the applications that you want to allow to preempt the kernel's better judgement.

Whitelisting

If you are doing pro (or hobbyist at the pro-level, or anything other than Audacity), then you need to manually whitelist any application dealing in low latency audio.

In casual conversations, the term “realtime” tends to mean near-instant processing of sound such that there is no noticeable delay. However, this is different from what a computer sees as realtime, so the term is abused. To keep “realtime” and “low latency” distinct from one another, Slackermedia uses “low latency” to describe what most people consider “close enough” to realtime audio as necessary, and “realtime” to only mean actual realtime processing, such as what might be needed in an active, in-production recording studio or the stock market.

To whitelist an application, use the setcap command as root. The first sensible application to whitelist is the one driving all of the pro audio tools in the first place: the JACK daemon,jackd:

su -c '/sbin/setcap cap_ipc_lock,cap_sys_nice=ep /path/to/jackd'

After jackd has been whitelisted, whitelist the other pro audio applications that you use:

  • Stand-alone synths, especially if they have inbuilt effect processors.
  • The DAW(s) that you use.
  • Loopers, beatboxes, drum machines, sequencers.
  • Ffmpeg, if you intend to do screen captures with it.

You do not need to worry about getting everything all at once; if something needs to be less nice with other processes, you'll hear it. You can whitelist it as needed, and the change will be immediate.

If you came here for a low latency kernel, that's it! you're done. You can skip the next section.

Realtime Kernel

Most people casually express a desire for a “realtime” kernel when they actually mean a low latency kernel, but sometimes realtime is actually necessary. For instance, multiple tracks with multiple overdub takes might require realtime response in order for the musician to keep tempo. If you find that low latency is not sufficient, you can build your own kernel with true realtime capabilities.

To build a realtime kernel, download two things;

  1. The kernel source code
  2. Realtime patches for the source code

Patching the Kernel

TL;DR
You only need to do this if you have a real need for literal realtime performance. This will make no difference to you if you are not using applications that take advantage of realtime capabilities, so don't bother doing this under the illusion that it will make your system faster for graphics editing or gaming.

The code to provide the Linux kernel with true realtime performance is maintained separately from the main kernel code itself. It is available from https://www.kernel.org/pub/linux/kernel/projects/rt, so take a look at the list of supported kernel versions and then download the latest stable version kernel that matches up with the available patches.

In other words, if the latest version number listed in the patches folder online is 4.0, then go into that folder and look at what patches are available. The patch files are labelled with the version of the kernel they support, so if there is a file, for instance, called patch-4.0.4-rt1.patch.xz then download that patch file and the 4.0.4 kernel.

Do not download the patches file (plural), but the patch file (singular). The “patches” file contains all the patches in separate documents, and will fail.

To download the kernel source code, go to http://kernel.org and download it. Alternately, you can download it using a shell based browser like wget:

$ wget http://kernel.org/pub/linux/kernel/vX.Y/linux-X.Y.ZZ.tar.bz2
$ wget https://kernel.org/pub/linux/kernel/projects/rt/X.Y/patch-X.Y.ZZ-rt1.patch.xz

You should now have two files:

  1. A single patch file for realtime.
  2. The Linux kernel source code.

Extract the kernel source and change into its directory:

# tar xvf linux-X.Y.ZZ.tar.bz2 -C /usr/src/linux-X.YY.ZZ
# cd /usr/src/linux-X.YY.ZZ

And then apply the realtime patch:

# xzcat ../patch-X.Y.ZZ-rt1.tar.xz | patch -p1

You will see a few screen-fulls of text and eventually get a BASH prompt back.

Now you can compile the new kernel.

Compiling the Linux Kernel

Configuring a kernel from scratch is a big job. It's arguably worth doing, but it requires solid knowledge about the hardware in your computer, down to the individual components. You only have to do the configuration once per computer, since you can inherit old configs and just add to them in the case of a new kernel.

Better yet, you can take advantage of the hard work that Slackware's maintainer, Patrick Volkerding, has done and inherit his configuration.

To do that, copy the Slackware kernel config file into the kernel sources folder. Find the kernel config files in /media/{cdrom}/testing/source/linux-X.X.XX.X-configs or on any Slackware mirror; for example, ftp://ftp.slackware.com/pub/slackware/slackware-XX/testing/source/linux-X.X.XX.X-configs/.

Presuming an SMP-capable CPU (you would know if your CPU is not SMP-capable), you will notice that there are two configuration files; the generic kernel configuration and the huge kernel configuration. By default, you are using the huge config, but this doesn't mean you have to keep using it. If you feel that your hardware is well-supported by the Linux kernel, then you may wish to try operating on a more lightweight kernel. You can always re-compile to a huge kernel if it turns out to be necessary.

# cp /media/cdrom/testing/source\
/linux-X.X.XX.X-configs/config-generic-smp-X.X.XX.X-smp \
/path/to/linux-X.X.XX.X/.config

Launch menuconfig, an ncurses-based, menu-driven list of possible options for your new kernel. The menu can be overwhelming, but as it starts off with the default Slackware configuration, there will not be much you need to change, but for the realtime options.

# make menuconfig

Most of the realtime options can be found in the Processor type and features menu.

Most default configurations here are sane, but confirm that the configuration is set to:

  • enable CONFIGPREEMPTRT
  • activate the High-Resolution-Timer Option
  • generally disable power management such as ACPI and APM. Not all ACPI options are bad, but cutting out as many as possible is best. The pmtimer option is required for a clock source; to activate pmtimer, activate ACPI_SUPPORT in menuconfig but deactivate all other sub modules like fan, processor or button.

Additionally, verify that your filesystem is supported in the configuration. Without the ability to read your harddrive, the kernel will not be able to load your operating system.

  1. Go into the File System menu.
  2. Activate the filesystem support you need, not as a module, but as code to compile into the kernel.

If you do not know what filesystem you are using, open another terminal and cat /etc/fstab. The filesystem (such as ext2, ext3, ext4, reiserfs, xfs, or jfs) is listed next to the partition label (/dev/sdX). If you are very advanced and are using two different types of filesystems for different partitions, then it is safest to include them both in your kernel.

Building and Installing the Kernel

With your new configuration file saved, compile the kernel and its modules.

# make bzImage
# make modules

Take a coffee break and come back when the compile is finished.

Once built, back up your current kernel image (you may need to boot back into it, if your attempt at building your own has failed).

# mv /boot/vmlinuz /boot/vmlinuz.stock
# mv /boot/System.map /boot/System.map.stock

And then move the new compressed kernel image (bzImage) and system map into your boot directory.

# cat arch/x86_64/boot/bzImage > /boot/vmlinuz
# cp System.map /boot/System.map

And finally install the kernel modules (the drivers):

# make modules_install

When this finishes, do not reboot.

Lilo

The new realtime kernel is now in place and its modules are installed, but the bootloader (the part of the system that gets loaded after the computer has been powered on but before an OS has been selected) needs to know where the new kernel is, and to use it. It also should be told where the last known good kernel is located so that if your new kernel fails for whatever reason, you can still boot your computer!

Using kdesu to grant you root privileges within your graphical environment (or you can work in the terminal if you are comfortable with terminal-based text editors), edit lilo.conf in the text editor of your choice;

$ kdesu emacs /etc/lilo.conf

In you text editor, duplicate the current entry, but change the label to reflect that it will boot the realtime kernel, and change the vmlinuz reference in the stock kernel so that it points to vmlinux.stock rather than vmlinuz. In the end, you will have entries quite similar to this:

image = /boot/vmlinuz
root = /dev/sda1
label = slackermedia
read-only

image = /boot/vmlinuz.stock
root = /dev/sda1
label = slackware
read-only

You may also change the delay that it takes for LILO to choose its default entry. By default, Slackware provides you two minutes (1200 milliseconds). Depending on your needs, a time of 30 or 60 milliseconds (that is, 3 or 6 seconds) may be sufficient.\

Update the bootloader with its new configuration:

# lilo

Reboot the machine with your new realtime kernel.

It should boot as quickly as or quicker than (depending on how much customization you did in menuconfig) the stock Slackware kernel.

To verify that you have the correct kernel running after a successful boot, check the dmesg output for the strings “Real-Time Preemption Support / Ingo Molnar”:

# dmesg | grep -i preemption

If the new kernel does not boot as expected, then it will most certainly provide errors during the boot process. Take note of the errors, reboot to with the stock kernel, and research what the errors mean.

Always remember to run lilo after doing a new kernel installation!

Typical errors are that you have left some important code to be built as modules rather than building them into the kernel. Often, this is the filesystem, so verify that you have the proper filesystem(s) included.

Another possibility is that you have removed some vital component of the kernel altogether in an effort to reduce its size. Research the issue, go back to the source code (in /usr/src/linux-XX.YY.ZZ) and run (as root) make clean ; make menuconfig and include what you had previously taken out of the kernel.

If you had installed Slackware with the huge kernel and are trying to use the generic configuration, then it's possible that you do need the huge kernel after all, meaning you'll need to download the huge configuration file and make that into the config file and compile again.

R S Q