Configuring and Building
the Kernel
The kernel is the third
element of embedded Linux. It is the component that is responsible for managing
resources and interfacing with hardware, and so affects almost every aspect of
your final software build. It is usually tailored to your particular hardware
configuration, although, as we saw in 2, All
About Bootloaders, device trees
allow you to create a generic kernel that is tailored to particular hardware by the contents of the device tree.
In this , we will look at
how to get a kernel for a board, and how to configure and compile it. We will
look again at bootstrap, this time focusing on the part the kernel plays. We
will also look at device drivers and how they pick up information from the
device tree.
In this , we will cover the following topics:
What does the kernel do?
Choosing a kernel.
Building the kernel.
Booting the kernel.
Porting Linux to a new board.
What does the kernel do?
Linux began in 1991, when
Linus Torvalds started writing an operating system for Intel 386- and 486-based
personal computers. He was inspired by the Minix operating system written by Andrew S. Tanenbaum four years earlier.
Linux differed in many ways from Minix; the main differences being that it was
a 32-bit virtual memory kernel and the code was open source, later released
under the GPL v2 license. He announced it on 25th August, 1991, on the comp.os.minix newsgroup in a famous post
that began with:
Hello everybody out there using minix—I'm doing a (free) operating
system (just a hobby, won't be big and professional like GNU) for 386(486) AT
clones. This has been brewing since April, and is starting to get ready. I'd
like any feedback on things people like/dislike in minix, as my OS resembles it
somewhat (same physical layout of the filesystem (due to practical reasons)
among other things).
To be strictly accurate,
Linus did not write an operating system, rather he wrote a kernel, which is
only one component of an operating system. To create a complete operating
system with user space commands and a shell command interpreter, he used
components from the GNU project, especially the toolchain, the C-library, and
basic command-line tools. That distinction remains today, and gives Linux a lot
of flexibility in the way it is used. It can be combined with a GNU user space
to create a full Linux distribution that runs on desktops and servers, which is
sometimes called GNU/Linux; it can be combined with an Android user space to
create the well-known mobile operating system, or it can be combined with a
small BusyBox-based user space to create a compact embedded system. Contrast
this with the BSD operating systems, FreeBSD, OpenBSD, and NetBSD, in which the
kernel, the toolchain, and the user space are combined into a single code base.
The kernel has three main
jobs: to manage resources, to interface with hardware, and to provide an API
that offers a useful level of abstraction to user space programs, as summarized
in the following diagram:
Applications running in User space run at a low CPU privilege
level. They can do very little other than make library calls. The primary
interface between the User space and
the Kernel space is the C library, which translates user level functions, such as those defined by
POSIX, into kernel system calls. The system call interface uses an
architecture-specific method, such as a trap or a software interrupt, to switch
the CPU from low privilege user mode to high privilege kernel mode, which
allows access to all memory addresses and CPU registers.
The System call handler dispatches the call to the appropriate kernel
subsystem: memory allocation calls go to the memory manager, filesystem calls
to the filesystem code, and so on. Some of those calls require input from the
underlying hardware and will be passed down to a device driver. In some cases,
the hardware itself invokes a kernel function by raising an interrupt.
The preceding diagram shows that there is a second entry point into
kernel code: hardware interrupts. Interrupts can only be handled in a device
driver, never by a user space application.
In other words, all the
useful things that your application does, it does them through the kernel. The
kernel, then, is one of the most important elements in the system.
Choosing a kernel
The next step
is to choose the kernel for your project, balancing the desire to always use
the latest version of software against the need for vendor-specific additions
and an interest in the long term support of the code base.
Kernel development cycle
Linux is developed at a fast pace, with a new
version being released every 8 to 12 weeks. The way that the version numbers
are constructed has changed a bit in recent years. Before July 2011, there was
a three number version scheme with version numbers that looked like 2.6.39. The
middle number indicated whether it was a developer or stable release; odd
numbers (2.1.x, 2.3.x, 2.5.x) were for developers and
even numbers were for end users. From version 2.6 onwards, the idea of a
long-lived development branch (the odd numbers) was dropped, as it slowed down
the rate at which new features were made available to the users. The change in
numbering from 2.6.39 to 3.0 in July 2011 was purely because Linus felt that
the numbers were becoming too large; there was no huge leap in the features or
architecture of Linux between those two versions. He also took the opportunity
to drop the middle number. Since then, in April 2015, he bumped the major from
3 to 4, again purely for neatness, not because of any large architectural
shift.
Linus manages the development kernel tree. You can follow him by cloning
the
Git tree like so:
$ git
clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
This will check out into
subdirectory linux. You can keep up to date
by running the command git pull
in that directory from time to time.
Currently, a full cycle of
kernel development begins with a merge window of two weeks, during which Linus
will accept patches for new features. At the end of the merge window, a
stabilization phase begins, during which Linus will produce weekly release
candidates with version numbers ending in -rc1, -rc2, and so on, usually up to -rc7 or -rc8. During this time, people
test the candidates and submit bug reports and fixes. When all significant bugs
have been fixed, the kernel is released.
The code
incorporated during the merge window has to be fairly mature already. Usually,
it is pulled from the repositories of the many subsystem and architecture
maintainers of the kernel. By keeping to a short development cycle, features
can
be merged when they are
ready. If a feature is deemed not sufficiently stable or well developed by the
kernel maintainers, it can simply be delayed until the next release.
Keeping a track of what has
changed from release to release is not easy. You can read the commit log in
Linus' Git repository but, with roughly 10,000 or more entries, it is not easy
to get an overview. Thankfully, there is the Linux Kernel Newbies website, http://kernelnewbies.org, where
you will find a succinct overview of each
version at http://kernelnewbies.org/LinuxVersions.
Stable and long term
support releases
The rapid rate of change of
Linux is a good thing in that it brings new features into the mainline code
base, but it does not fit very well with the longer life cycle of embedded
projects. Kernel developers address this in two ways, with stable releases and long
term releases. After the release of a mainline kernel (maintained by Linus Torvalds) it is moved to the stable tree (maintained by Greg
Kroah-Hartman). Bug fixes are applied to the stable kernel, while the mainline
kernel begins the next development cycle. Point releases of the stable kernel
are marked by a third number, 3.18.1, 3.18.2, and so on. Before version 3,
there were four release numbers, 2.6.29.1, 2.6.39.2, and so on.
You can get the stable tree by using the following command:
$ git
clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
You can use git checkout to get a particular version, for example
version 4.9.13:
$ cd
linux-stable
$ git
checkout v4.9.13
Usually, the stable kernel is updated only until
the next mainline release (8 to 12 weeks later), so you will see that there is
just one or sometimes two stable kernels at https://www.kernel.org/. To cater for those users
who would like updates for a longer period of time and be assured that any bugs
will be found and fixed, some kernels are labeled long term and maintained for two or more years. There is at least
one long term kernel release each year. Looking at https://www.kernel.org/ at the time of writing, there are a total of nine long term kernels:
4.9, 4.4, 4.1, 3.18, 3.14, 3.12, 3.10, 3.4, and 3.2. The latter has been
maintained for five years and is at version 3.2.86. If you are building a
product that you will have to maintain for this length of time, then the latest
long term kernel might well be a good choice.
Vendor support
In an ideal
world, you would be able to download a kernel from https://www.kernel.or g/ and configure it for any
device that claims to support Linux. However, that is not always possible; in fact mainline Linux has solid support for
only a small subset of the many devices that can run Linux. You may find
support for your board or SoC from independent open source projects, Linaro or
the Yocto Project, for example, or from companies providing third party support
for embedded Linux, but in many cases you will be obliged to look to the vendor
of your SoC or board for a working kernel. As we know, some are better at
supporting Linux than others. My only advice at this point is to choose vendors
who give good support or who, even better, take the trouble to get their kernel
changes into the mainline.
Licensing
The Linux source code is
licensed under GPL v2, which means that you must make the source code of your
kernel available in one of the ways specified in the license.
The actual text of the
license for the kernel is in the file COPYING. It begins with an addendum
written by Linus that states that code calling the kernel from user space via
the system call interface is not considered a derivative work of the kernel and
so is not covered by the license. Hence, there is no problem with proprietary
applications running on top of Linux.
However, there is one area
of Linux licensing that causes endless confusion and debate: kernel modules. A
kernel module is simply a piece of code that is dynamically linked with the
kernel at runtime, thereby extending the functionality of the kernel. The GPL
makes no distinction between static and dynamic linking, so it would appear
that the source for kernel modules is covered by the GPL. But, in the early
days of Linux, there were debates about exceptions to this rule, for example,
in connection with the Andrew filesystem. This code predates Linux and
therefore (it was argued) is not a derivative work, and so the license does not
apply. Similar discussions took place over the years with respect to other
pieces of code, with the result that it is now accepted practice that the GPL
does not necessarily apply to kernel
modules. This is codified by the kernel MODULE_LICENSE macro, which may take the value Proprietary to indicate that it is not
released under the GPL. If you plan to use the same arguments yourself, you may
want to read though an oft-quoted e-mail thread titled "Linux GPL and binary
module exception clause?" which is archived at http ://yarchive.net/comp/linux/gpl_modules.html.
The GPL should be
considered a good thing because it guarantees that when you and I are working
on embedded projects, we can always get the source code for the kernel. Without
it, embedded Linux would be much harder to use and more fragmented.
Building the kernel
Having decided which kernel to base your build on, the next step is to build
it.
Getting the source
Both of the targets used in
this , the BeagleBone Black and the ARM Versatile PB, are well supported by the
mainline kernel. Therefore, it makes sense to use the latest long-term kernel
available from https://www.kernel.org/, which at the time of
writing was 4.9.13. When you come to do this for yourself, you should check to
see if there is a later version of the 4.9 kernel and use that instead since it
will have fixes for bugs found after 4.9.13 was released. If there is a later
long-term release, you may want to consider using that one, but be aware that
there may have been changes that mean that the following sequence of commands
do not work exactly as given.
Use this command to clone the stable kernel and check out version
4.9.13:
$ git
clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
$ cd
linux-stable
$ git
checkout v4.9.13
There is a lot of code
here. There are over 57,000 files in the 4.9 kernel containing C-source code,
header files, and assembly code, amounting to a total of over 14 million lines
of code, as measured by the SLOCCount utility. Nevertheless, it is worth
knowing the basic layout of the code and to know, approximately, where to look
for a particular component. The main directories of interest are:
arch: Contains
architecture-specific files. There is one subdirectory per architecture.
Documentation: Contains kernel
documentation. Always look here first if you want to find more information about an aspect of
Linux.
drivers: Contains device drivers,
thousands of them. There is a subdirectory for each type of driver.
fs: Contains filesystem code.
include: Contains kernel header
files, including those required when building the toolchain.
kernel: Contains core functions,
including scheduling, locking, timers, power management, and debug/trace code. mm: Contains memory
management.
net: Contains network protocols.
scripts: Contains many useful
scripts, including the device tree compiler, DTC, which I described in 3, All
About Bootloaders.
tools: Contains many useful
tools and including the Linux performance counters tool, perf, which I will describe in 15, Profiling
and Tracing.
Over a period of time, you
will become familiar with this structure, and realize that if you are looking
for the code for the serial port of a particular SoC, you will find it in drivers/tty/serial and not in arch/$ARCH/mach-foo, because it is a device
driver and not something central to the running of Linux on that SoC.
Understanding kernel configuration – Kconfig
One of the strengths of
Linux is the degree to which you can configure the kernel to suit different
jobs, from a small dedicated device such as a smart thermostat to a complex
mobile handset. In current versions, there are many thousands of configuration
options. Getting the configuration right is a task in itself but, before that,
I want to show you how it works so that you can better understand what is going
on.
The configuration mechanism is called Kconfig, and the build system that
it integrates with is called Kbuild.
Both are documented in Documentation/kbuild. Kconfig/Kbuild
is used in
a number of other projects as well as the kernel, including Crosstool-NG, U-Boot,
Barebox, and BusyBox.
The configuration options are declared in a hierarchy of files named Kconfig,
using a syntax described in Documentation/kbuild/kconfig-language.txt. In Linux, the
top level Kconfig looks like this:
mainmenu "Linux/$ARCH $KERNELVERSION Kernel
Configuration"
config SRCARCH
string
option env="SRCARCH"
source "arch/$SRCARCH/Kconfig"
The last line includes the
architecture-dependent configuration file which sources other Kconfig files, depending on which
options are enabled. Having the architecture play such a role has two
implications: firstly, that you must specify an architecture when configuring
Linux by setting ARCH=[architecture], otherwise it will default
to the local machine architecture, and second, that the layout of the top level
menu is different for each architecture.
The value you put into ARCH is one of the subdirectories you find in
directory arch,
with the oddity that ARCH=i386 and ARCH=x86_64 both source arch/x86/Kconfig.
The Kconfig files consist largely of menus, delineated by menu and endmenu
taken from drivers/char/Kconfig:
menu "Character devices"
[...]
config DEVMEM
bool "/dev/mem virtual device support"
default y
help
Say Y here if
you want to support the /dev/mem device. The /dev/mem device is used to access
areas of physical memory.
When in doubt, say "Y".
[...]
endmenu
The parameter following config names a variable that, in
this case, is DEVMEM. Since this option is a bool (Boolean), it can only
have two values: if it is enabled, it is assigned to y, if it is not enabled, the
variable is not defined at all. The name of the menu item that is displayed on
the screen is the string following the bool keyword.
This configuration item,
along with all the others, is stored in a file named .config (note that the leading dot
(.) means that it is a hidden
file that will not be shown by the ls command, unless you type ls -a to show all the files).
The line corresponding to this configuration item reads:
CONFIG_DEVMEM=y
There are several other data types in addition to bool. Here is the list:
bool: Either y or not defined.
tristate: Used where a feature can
be built as a kernel module or built into the main kernel image. The values are m for a module, y to be built in, and not
defined if the feature is not enabled.
int: An integer value using decimal notation.
hex: An unsigned integer value using hexadecimal notation.
string: A string value.
There may be dependencies
between items, expressed by the depends on
construct, as shown here:
config MTD_CMDLINE_PARTS
tristate "Command line partition table
parsing"
depends on MTD
If
CONFIG_MTD has not been enabled
elsewhere, this menu option is not shown and so cannot be selected.
There are also reverse
dependencies; the select keyword enables other
options if this one is enabled. The Kconfig file in arch/$ARCH has a large number of select statements that enable
features specific to the architecture, as can be seen here for ARM:
config ARM
bool
default y
select ARCH_CLOCKSOURCE_DATA
select ARCH_HAS_DEVMEM_IS_ALLOWED
[...]
There are several configuration utilities that
can read the Kconfig files and produce a .config file. Some of them display
the menus on screen and allow you to make choices interactively. menuconfig is probably the one most
people are familiar with, but there are also xconfig and gconfig.
You launch each one via the
make command, remembering that,
in the case of the kernel, you have to supply an architecture, as illustrated
here:
$ make
ARCH=arm menuconfig
Here, you can see menuconfig with the DEVMEM config option highlighted
in the previous paragraph:
The star (*) to the left of an item means that it is selected (Y) or, if
it is an M,
You often see instructions like enable CONFIG_BLK_DEV_INITRD, but with so many menus to browse through, it can take a while to find
the place where that configuration is set. All configuration editors have a
search function. You can access it in menuconfig by pressing the forward slash key, /. In xconfig, it is in the edit menu, but make sure you miss off CONFIG_ part of the configuration item
you are searching for.
With so many things to
configure, it is unreasonable to start with a clean sheet each time you want to
build a kernel, so there are a set of known working configuration files in arch/$ARCH/configs, each containing suitable
configuration values for a single SoC or a group of SoCs.
You can select
one with the make
[configuration file name] command. For example, to configure Linux to run on a wide range of SoCs
using the ARMv7-A architecture, you would
type:
$ make
ARCH=arm multi_v7_defconfig
This is a generic kernel
that runs on various different boards. For a more specialized application, for
example, when using a vendor-supplied kernel, the default configuration file is
part of the board support package; you will need to find out which one to use
before you can build the kernel.
There is another useful
configuration target named oldconfig.
This takes an existing .config
file and asks you to supply configuration values for any options that don't
have them. You would use it when moving a configuration to a newer kernel
version; copy .config from the old kernel to the
new source directory and run the make ARCH=arm oldconfig command to bring it up to
date. It can also be used to validate a .config file that you have edited manually (ignoring
the text ""Automatically
generated file; DO NOT EDIT"" that occurs at the top; sometimes
it is OK to ignore warnings).
If you do make
changes to the configuration, the modified .config file becomes part of your board support package
and needs to be placed under source code control.
Using LOCALVERSION to identify your kernel
You can discover the kernel version that you have built using the make
kernelversion target:
$ make
ARCH=arm kernelversion
4.9.13
This is reported at runtime
through the uname command, and is also used
in naming the directory where kernel modules are stored.
If you change
the configuration from the default, it is advisable to append your own version
information, which you can configure by setting CONFIG_LOCALVERSION. As an example, if I
wanted to mark the kernel I am building with the identifier melp and version 1.0, I would
define the local version in menuconfig like this:
Running make
kernelversion produces the
same output as before, but now if I run
make kernelrelease, I see:
$ make
ARCH=arm kernelrelease
4.9.13-melp-v1.0
Kernel modules
I have mentioned kernel
modules several times already. Desktop Linux distributions use them extensively
so that the correct device and kernel functions can be loaded at runtime,
depending on the hardware detected and features required. Without them, every
single driver and feature would have to be statically linked in to the kernel,
making it infeasibly large.
On the other hand, with
embedded devices, the hardware and kernel configuration is usually known at the
time the kernel is built, and therefore modules are not so useful. In fact,
they cause a problem because they create a version dependency between the
kernel and the root filesystem, which can cause boot failures if one is updated
but not the other. Consequently, it is quite common for embedded kernels to be
built without any modules at all.
Here are a few cases where
kernel modules are a good idea in embedded systems:
When you have proprietary
modules, for the licensing reasons given in the preceding section.
To reduce boot time by
deferring the loading of non-essential drivers. When there are a number of
drivers that could be loaded and it would take up too much memory to compile
them statically. For example, you have a USB interface that supports a range of
devices. This is essentially the same argument as is used in desktop
distributions.
Compiling – Kbuild
The kernel build system, Kbuild, is a set of make scripts that take the
configuration information from the .config file, work out the dependencies, and compile
everything that is necessary to produce a kernel image containing all the
statically linked components, possibly a device tree binary and possibly one or
more kernel modules. The dependencies are expressed in makefiles that are in each directory
with buildable components. For instance, the following two lines are taken from
drivers/char/Makefile:
obj-y += mem.o random.o
obj-$(CONFIG_TTY_PRINTK) += ttyprintk.o
The obj-y rule unconditionally
compiles a file to produce the target, so mem.c and random.c are always part of the kernel. In the second
line,
ttyprintk.c is
dependent
on a
configuration parameter. If CONFIG_TTY_PRINTK is y, it is compiled as a
built-in; if it is m, it is built as a module;
and if the parameter is undefined, it is not compiled at all.
For most targets, just
typing make (with the appropriate ARCH and CROSS_COMPILE) will do the job, but it
is instructive to take it one step at a time.
Finding out which kernel target to build
To build a kernel image,
you need to know what your bootloader expects. This is a rough guide:
U-Boot:
Traditionally, U-Boot has required uImage, but newer versions can load a zImage file using the bootz command x86
targets: Requires a bzImage file
Most
other bootloaders:
Require a zImage file
Here is an example of building a zImage file:
$ make
-j 4 ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- zImage
The -j 4 option tells make how many jobs to run in parallel, which reduces the time taken to
build. A rough guide is to run as many jobs as you have CPU cores.
There is a small issue with
building a uImage file for ARM with
multi-platform support, which is the norm for the current generation of ARM SoC
kernels. Multi-platform support for ARM was introduced in Linux 3.7. It allows
a single kernel binary to run on multiple platforms and is a step on the road
toward having a small number of kernels for all ARM devices. The kernel selects
the correct platform by reading the machine number or the device tree passed to
it by the bootloader. The problem occurs because the location of physical
memory might be different for each platform, and so the relocation address for
the kernel (usually 0x8000
bytes from the start of physical RAM) might also be different. The relocation
address is coded into the uImage
header by the mkimage command when the kernel is
built, but it will fail if there is more than one relocation address to choose
from. To put it another way, the uImage format is not compatible with multi-platform
images. You can still create a uImage
binary from a multi-platform build, so long as you give the LOADADDR of the particular SoC you
are hoping to boot this kernel on. You can find the load address by looking in mach - [your SoC]/Makefile.boot and noting the value of zreladdr-y:
Build artifacts
A kernel build generates
two files in the top level directory: vmlinux and System.map. The first, vmlinux, is the kernel as an ELF binary. If you have
compiled your kernel with debug enabled (CONFIG_DEBUG_INFO=y), it will contain debug
symbols which can be used with debuggers like kgdb. You can also use other ELF binary tools, such
as size:
$
arm-cortex_a8-linux-gnueabihf-size vmlinux
text data bss
dec hex filename 10605896 5291748 351864 16249508 f7f2a4 vmlinux
System.map contains the
symbol table in a human readable form.
Most bootloaders cannot
handle ELF code directly. There is a further stage of processing which takes vmlinux and places those binaries
in arch/$ARCH/boot that are suitable for the
various bootloaders:
Image: vmlinux converted to raw binary format.
zImage: For the PowerPC
architecture, this is just a compressed version of Image, which implies that the
bootloader must do the decompression. For all other architectures, the compressed Image is piggybacked onto a stub
of code that decompresses and relocates it.
uImage: zImage plus a 64-byte U-Boot header.
While the build is running,
you will see a summary of the commands being executed:
$ make -j 4 ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- \ zImage
CHK
include/config/kernel.release
CHK
include/generated/uapi/linux/version.h HOSTCC scripts/basic/fixdep
HOSTCC
scripts/kallsyms HOSTCC scripts/dtc/dtc.o [...]
Sometimes, when the kernel
build fails, it is useful to see the actual commands being executed. To do
that, add V=1 to the command line:
$ make -j 4 ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- \ V=1
zImage
Compiling device trees
The next step is to build the device tree, or trees if you have a
multi-platform
build. The dtbs target builds device trees according to the
rules in
arch/$ARCH/boot/dts/Makefile, using the device tree source files in that directory.
Following is a snippet from building the dtbs target for multi_v7_defconfig:
$ make ARCH=arm dtbs [...]
DTC arch/arm/boot/dts/alpine-db.dtb
DTC
arch/arm/boot/dts/artpec6-devboard.dtb DTC arch/arm/boot/dts/at91-kizbox2.dtb
DTC
arch/arm/boot/dts/at91-sama5d2_xplained.dtb DTC
arch/arm/boot/dts/at91-sama5d3_xplained.dtb DTC
arch/arm/boot/dts/sama5d31ek.dtb
[...]
The compiled .dtb files are generated in the same directory as
the sources.
Compiling modules
If you have configured some
features to be built as modules, you can build them separately using the modules target:
$ make -j 4 ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- \
modules
The compiled modules have a .ko suffix and are generated
in the same directory as the source code, meaning that they are scattered all
around the kernel source tree. Finding them is a little tricky, but you can use
the modules_install
make target to
install them in the right place. The default location is /lib/modules in your development
system, which is almost certainly not what you want. To install them into the
staging area of your root filesystem (we will talk about root filesystems in
the next ), provide the path using INSTALL_MOD_PATH:
$ make -j4 ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- \
INSTALL_MOD_PATH=$HOME/rootfs modules_install
Kernel modules are put into
the directory /lib/modules/[kernel
version],
relative to the root of the filesystem.
Cleaning kernel sources
There are three make targets for cleaning the kernel source tree:
clean: Removes object files and most intermediates.
mrproper: Removes all intermediate
files, including the .config file.
Use this
target to
return the source tree to the state it was in immediately after cloning or
extracting the source code. If you are curious about the name, Mr Proper is a
cleaning product common in some parts of the world. The meaning of make mrproper is to give the kernel
sources a really good scrub. distclean:
This is the same as
mrproper,
but also deletes editor backup files, patch files, and other artifacts of software
development.
Building a kernel for the BeagleBone Black
In light of the information
already given, here is the complete sequence of commands to build a kernel, the
modules, and a device tree for the BeagleBone Black, using the Crosstool-NG ARM
Cortex A8 cross compiler:
$ cd
linux-stable
$ make
ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- mrproper
$ make
ARCH=arm multi_v7_defconfig
$ make -j4
ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- zImage
$ make -j4
ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- modules
$ make
ARCH=arm CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf- dtbs
These commands are in the script MELP/_04/build-linux-bbb.sh .
Building a kernel for QEMU
Here is the sequence of
commands to build Linux for the ARM Versatile PB that is emulated by QEMU,
using the Crosstool-NG V5te compiler:
$ cd
linux-stable
$ make
ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- mrproper
$ make -j4
ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- zImage
$ make -j4
ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- modules
$ make
ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- dtbs
These commands are in the script MELP/_04/build-linux-versatilepb.sh.
Booting the kernel
Booting Linux is highly
device-dependent. In this section, I will show you how it works for the
BeagleBone Black and QEMU. For other target boards, you must consult the
information from the vendor or from the community project, if there is one.
At this point, you should
have the zImage file and the dtbs targets for the BeagleBone
Black or QEMU.
Booting the BeagleBone
Black
To begin, you need a
microSD card with U-Boot installed, as described in the section Installing U-Boot. Plug the microSD card
into your card reader and from the linux-stable directory the files arch/arm/boot/zImage and arch/arm/boot/dts/am335x-
boneblack.dtb to the boot partition. Unmount the card
and plug it into the BeagleBone
Black. Start a terminal emulator, such as gtkterm, and be prepared to press the space bar as soon
as you see the U-Boot messages appear. Next, power on the BeagleBone Black and
press the space bar. You should get a U-Boot prompt, . Now enter the following
commands to load Linux and the device tree binary:
U-Boot#
fatload mmc 0:1 0x80200000 zImage
reading zImage
7062472 bytes read in 447 ms (15.1 MiB/s)
U-Boot#
fatload mmc 0:1 0x80f00000 am335x-boneblack.dtb
reading am335x-boneblack.dtb
34184 bytes read in 10 ms (3.3 MiB/s)
U-Boot#
setenv bootargs console=ttyO0
U-Boot#
bootz 0x80200000 - 0x80f00000
## Flattened Device Tree blob
at 80f00000 Booting using the fdt blob at 0x80f00000
Loading Device Tree to 8fff4000, end 8ffff587
... OK
Starting kernel ...
[ 0.000000] Booting Linux on
physical CPU 0x0 [...]
Note that we set the kernel
command line to console=ttyO0. That tells Linux which
device to use for console output, which in this case is the first UART on the
board, device ttyO0. Without this, we would
not see any messages after Starting
the kernel..., and therefore would not
know if it was working or not. The sequence will end in a kernel panic, for reasons
I will explain later on.
Booting QEMU
Assuming that you have
already installed qemu-system-arm, you can launch with the
kernel and the .dtb file for the ARM Versatile
PB, as follows:
$
QEMU_AUDIO_DRV=none \
qemu-system-arm
-m 256M -nographic -M versatilepb -kernel zImage \
-append
"console=ttyAMA0,115200" -dtb versatile-pb.dtb
Note that setting QEMU_AUDIO_DRV to none is just to suppress error
messages from QEMU about missing configurations for the audio drivers, which we
do not use. As with the BeagleBone Black, this will end with a kernel panic and
the system will halt. To exit from QEMU, type Ctrl + A and then x (two separate keystrokes).
Kernel panic
While things started off well, they ended badly:
[ 1.886379] Kernel panic - not syncing: VFS:
Unable to mount root fs on unknown-block(0
[ 1.895105] ---[ end Kernel panic - not syncing:
VFS: Unable to mount root fs on unknow
This is a good example of a
kernel panic. A panic occurs when the kernel encounters an unrecoverable error.
By default, it will print out a message to the console and then halt. You can
set the panic command-line parameter to
allow a few seconds before reboots following a panic. In this case, the
unrecoverable error is no root filesystem, illustrating that a kernel is
useless without a user space to control it. You can supply a user space by
providing a root filesystem, either as a ramdisk or on a mountable mass storage
device. We will talk about how to create a root filesystem in the next , but
first I want to describe the sequence of events that leads up to panic.
Early user space
In order to transition from
kernel initialization to user space, the kernel has to mount a root filesystem
and execute a program in that root filesystem. This can be achieved via a
ramdisk or by mounting a real filesystem on a block device. The code for all of
this is in init/main.c, starting with the
function rest_init(), which creates the first
thread with PID 1 and runs the code in kernel_init(). If there is a ramdisk, it
will try to execute the program /init,
which will take on the task of setting up the user space.
If fails to find and run /init, it tries to mount a filesystem by calling the
function
prepare_namespace() in init/do_mounts.c. This requires a root= command line to give
the name of the block device to use for mounting, usually in the form:
root=/dev/<disk name><partition
number>
Or, for SD cards and eMMC:
root=/dev/<disk name>p<partition
number>
For example, for the first
partition on an SD card, that would be root=/dev/mmcblk0p1. If the mount succeeds, it
will try to execute
/sbin/init, followed by /etc/init, /bin/init, and then /bin/sh, stopping at the first one
that works.
The program can be
overridden on the command line. For a ramdisk, use rdinit=, and for a filesystem, use
init=.
Kernel messages
Kernel developers are fond
of printing out useful information through liberal use of printk() and similar functions. The
messages are categorized according to importance, with 0 being the highest:
|
Level
|
Value
|
Meaning
|
|
|
|
|
|
KERN_EMERG
|
0
|
The system is unusable
|
|
|
|
|
|
KERN_ALERT
|
1
|
Action must be taken
immediately
|
|
|
|
|
|
KERN_CRIT
|
2
|
Critical conditions
|
|
|
|
|
|
KERN_ERR
|
3
|
Error conditions
|
|
|
|
|
|
KERN_WARNING
|
4
|
Warning conditions
|
|
|
|
|
|
KERN_NOTICE
|
5
|
Normal but significant
conditions
|
|
|
|
|
|
KERN_INFO
|
6
|
Informational
|
|
|
|
|
|
KERN_DEBUG
|
7
|
Debug-level messages
|
|
|
|
|
They are first written to a buffer, __log_buf, the size of
which is two to the power
of CONFIG_LOG_BUF_SHIFT. For example, if CONFIG_LOG_BUF_SHIFT is 16, then __log_buf is 64
KiB. You can dump the entire buffer using the command dmesg.
If the level of
a message is less than the console log level, it is displayed on the console as
well as placed in __log_buf. The default console log
level is 7, meaning that messages of
level 6 and lower are displayed,
filtering out KERN_DEBUG, which is level 7. You can change the
console log level in several ways, including by using the kernel parameter loglevel=<level>, or the command dmesg -n <level>.
Kernel command line
The kernel command line is
a string that is passed to the kernel by the bootloader, via the bootargs variable in the case of
U-Boot; it can also be defined in the device tree, or set as part of the kernel
configuration in CONFIG_CMDLINE.
We have seen
some examples of the kernel command line already, but there are many more.
There is a complete list in Documentation/kernel-parameters.txt. Here is a smaller list of the most useful
ones:
|
Name
|
Description
|
|
|
|
|
|
|
debug
|
Sets the console log level
to the highest level, 8, to ensure that you
|
|
|
see all the kernel messages on the console.
|
||
|
|
||
|
|
|
|
|
init=
|
The init program to run
from a mounted root filesystem, which
|
|
|
defaults to /sbin/init.
|
||
|
|
||
|
|
|
|
|
lpj=
|
Sets loops_per_jiffy to a given constant. There
is a description of the
|
|
|
significance of this in the paragraph
following this table.
|
||
|
|
||
|
|
|
|
|
|
Behavior when the kernel
panics: if it is greater than zero, it gives
|
|
|
panic=
|
the number of seconds before rebooting; if it
is zero, it waits
|
|
|
forever (this is the
default); or if it is less than zero, it reboots
|
||
|
|
||
|
|
without any delay.
|
|
|
|
|
|
|
|
Sets the console log level to , suppressing
all but emergency
|
|
|
quiet
|
messages. Since most
devices have a serial console, it takes time
|
|
|
to
output all those strings. Consequently, reducing the number of
|
||
|
|
||
|
|
messages using this option reduces boot time.
|
|
|
|
|
|
|
rdinit=
|
The init program to run from a ramdisk. It
defaults to /init.
|
|
|
|
|
|
|
ro
|
Mounts the root device as
read-only. Has no effect on a ramdisk,
|
|
|
which is always read/write.
|
||
|
|
||
|
|
|
|
|
root=
|
Device to mount the root filesystem.
|
|
|
|
|
|
|
|
The number of seconds to
wait before trying to mount the root
|
|
|
rootdelay=
|
device;
defaults to zero. Useful if the device takes time to probe
|
|
|
|
the hardware, but also see rootwait.
|
|
|
|
|
The filesystem
type for the root device. In many cases, it is auto-rootfstype= detected during mount, but
it is required for jffs2 filesystems.
Waits indefinitely for the root device to be
detected. Usually
rootwait
necessary with mmc devices.
|
rw
|
Mounts the root device as
read-write (default).
|
The lpj parameter is often
mentioned in connection with reducing the kernel boot time. During
initialization, the kernel loops for approximately 250 ms to calibrate a delay
loop. The value is stored in the variable loops_per_jiffy, and reported like this:
Calibrating delay loop... 996.14 BogoMIPS
(lpj=4980736)
If the kernel always runs
on the same hardware, it will always calculate the same value. You can shave
250 ms off the boot time by adding lpj=4980736 to the command line.
Porting Linux to a new
board
Porting Linux to a new
board can be easy or difficult, depending on how similar your board is to an
existing development board. In 3, All About Bootloaders, we
ported U-Boot to a new board, named Nova, which is based on the BeagleBone Black. very few changes
to be made to the kernel code and so it very easy. If you are porting to
completely new and innovative hardware, there will be more to do. I am only
going to consider the simple case.
The organization of
architecture-specific code in arch/$ARCH
differs from one system to another. The x86 architecture is pretty clean
because most hardware details are detected at runtime. The PowerPC architecture
puts SoC and board-specific files into sub directory platforms. The ARM
architecture, on the other hand, is quite messy, in part because there is a lot
of variability between the many ARM-based SoCs. Platform-dependent code is put
in directories named mach-*,
approximately one per SoC. There are other directories named plat-* which contain code common
to several versions of an SoC. In the case of the BeagleBone Black, the
relevant directory is arch/arm/mach-omap2. Don't be fooled by the name though; it contains support for OMAP2, 3,
and 4 chips, as well as the AM33xx family of chips that the BeagleBone uses.
In the following sections,
I am going to explain how to create a device tree for a new board and how to
key that into the initialization code of Linux.
A new device tree
The first thing to do is
create a device tree for the board and modify it to describe the additional or
changed hardware of the Nova
board. In this simple case, we will just copy am335x-boneblack.dts to nova.dts and change the board name
in nova.dts, as shown highlighted
here:
/dts-v1/;
#include "am33xx.dtsi"
#include "am335x-bone-common.dtsi"
#include <dt-bindings/display/tda998x.h>
/ {
model =
"Nova";
compatible = "ti,am335x-bone-black",
"ti,am335x-bone", "ti,am33xx";
};
[...]
We can build the Nova device tree binary explicitly like this:
$ make
ARCH=arm nova.dtb
If we want the device tree for Nova to be
compiled by make
ARCH=arm dtbs
whenever an AM33xx target is selected, we could add a dependency in
arch/arm/boot/dts/Makefile as follows:
[...]
dtb-$(CONFIG_SOC_AM33XX) +=
nova.dtb
[...
We can see the
effect of using the Nova device tree by booting the BeagleBone Black, following
the same procedure as in the section Booting
the BeagleBone Black, with the
same zImage file as before, but loading nova.dtb in place of am335x-boneblack.dtb. The following highlighted
output is the point at which the machine model is printed out:
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.13-melp-v1.0-dirty
(chris@chris-xps) (gcc version 5.2.0 (
[ 0.000000] CPU: ARMv7 Processor [413fc082] revision
2 (ARMv7), cr=10c5387d
[ 0.000000] OF: fdt:Machine model: Nova [...]
Now that we have a device
tree specifically for the Nova board, we could modify it to describe the
hardware differences between Nova and the BeagleBone Black. There are quite
likely to be changes to the kernel configuration as well, in which case you
would create a custom configuration file based on a copy of arch/arm/configs/multi_v7_defconfig.
Setting the board compatible property
Creating a new
device tree means that we can describe the hardware on the Nova board,
selecting device drivers and setting properties to match. But, suppose the Nova
board needs different early initialization code than the BeagleBone Black; how
can we link that in?
The board setup is
controlled by the compatible property in the root node.
This is what we have for the Nova board at the moment:
/ {
model = "Nova";
compatible = "ti,am335x-bone-black",
"ti,am335x-bone", "ti,am33xx";
};
When the kernel parses this
node, it will search for a matching machine for each of the values of the
compatible property, starting on the left and stopping with the first match
found. Each machine is defined in a structure delimited by
DT_MACHINE_START and MACHINE_END macros. In
arch/arm/mach-omap2/board-generic.c, we
find:
#ifdef CONFIG_SOC_AM33XX
static const
char *const am33xx_boards_compat[]
__initconst = { "ti,am33xx",
NULL,
};
DT_MACHINE_START(AM33XX_DT, "Generic AM33XX
(Flattened Device Tree)")
.reserve = omap_reserve,
.map_io = am33xx_map_io,
.init_early = am33xx_init_early,
.init_machine = omap_generic_init,
.init_late = am33xx_init_late,
.init_time = omap3_gptimer_timer_init,
.dt_compat
= am33xx_boards_compat,
.restart = am33xx_restart,
MACHINE_END
#endif
Note that the
string array, am33xx_boards_compat , contains "ti,am33xx" which matches one of the
machines listed in the compatible property. In fact, it is the only match
possible, since there are none for ti,am335x-bone-black or ti,am335x-bone. The
structure between DT_MACHINE_START and MACHINE_END contains a pointer to the
string array, and function pointers for the board setup functions. You may
wonder why bother with ti,am335x-bone-black and ti,am335x-bone if they never match
anything? The answer is partly that they are place holders for the future, but
also that there are places in the kernel that contain runtime tests for the
machine using the
function of_machine_is_compatible(). For example, in drivers/net/ethernet/ti/cpsw-common.c:
int ti_cm_get_macid(struct device *dev, int
slave, u8 *mac_addr)
{
[...]
if
(of_machine_is_compatible("ti,am33xx"))
return cpsw_am33xx_cm_get_macid(dev, 0x630,
slave, mac_addr);
[...]
Thus, we have
to look through not just the mach-*
directories but the entire kernel source code to get a list of all the places
that depend on the machine compatible property. In the 4.9 kernel, you will
find that there are still no checks for
ti,am335x-bone-black and ti,am335x-bone, but there may be in the future.
Returning to the Nova board, if we want to add machine specific setup,
we can
add a machine in arch/arm/mach-omap2/board-generic.c, like this:
#ifdef CONFIG_SOC_AM33XX
[...]
static const
char *const nova_compat[] __initconst = { "ti,nova",
NULL,
};
DT_MACHINE_START(NOVA_DT, "Nova board
(Flattened Device Tree)")
.reserve = omap_reserve,
.map_io = am33xx_map_io,
.init_early = am33xx_init_early,
.init_machine = omap_generic_init,
.init_late = am33xx_init_late,
.init_time = omap3_gptimer_timer_init,
.dt_compat
= nova_compat,
.restart = am33xx_restart,
MACHINE_END
#endif
Then we could change the device tree root node like this:
/ {
model = "Nova";
compatible = "ti,nova",
"ti,am33xx";
};
Now, the machine will match ti,nova in board-generic.c. We keep ti,am33xx because
we
want the runtime tests, such as the one in drivers/net/ethernet/ti/cpsw-common.c, to continue to work.
TLDR;
Linux is a very powerful
and complex operating system kernel that can be married to various types of
user space, ranging from a simple embedded device, through increasingly complex
mobile devices using Android, to a full server operating system. One of its
strengths is the degree of configurability. The definitive place to get the
source code is https://www.kernel.org/, but you will probably
need to get the source for a particular SoC or board from the vendor of that
device or a third-party that supports that device. The customization of the
kernel for a particular target may consist of changes to the core kernel code,
additional drivers for devices that are not in mainline Linux, a default kernel
configuration file, and a device tree source file.
Normally, you start with
the default configuration for your target board, and then tweak it by running
one of the configuration tools such as menuconfig. One of the things you should consider at this
point is whether the kernel features and drivers should be compiled as modules
or built-in. Kernel modules are usually no great advantage for embedded
systems, where the feature set and hardware are usually well defined. However,
modules are often used as a way to import proprietary code into the kernel, and
also to reduce boot time by loading non-essential drivers after boot.
Building the kernel
produces a compressed kernel image file, named zImage , bzImage , or uImage, depending on the bootloader you will be using
and the target
architecture.
A kernel build will also generate any kernel modules (as .ko files) that you have
configured, and device tree binaries (as .dtb files) if your target requires them.
Porting Linux to a new target board can be quite
simple or very difficult, depending on how different the hardware is from that
in the mainline or vendor supplied kernel. If your hardware is based on a
well-known reference design, then it may be just a question of making changes
to the device tree or to the platform data.
The kernel is the core of a
Linux-based system, but it cannot work by itself. It requires a root filesystem
that contains the user space components. The root filesystem can be a ramdisk
or a filesystem accessed via a block device, which will be the subject of the
next . As we have seen, booting a kernel without a root filesystem results in a
kernel panic.
Build the compressed kernel image.. Thanks
ReplyDeleteKernel command line was helpful
ReplyDelete