The Linux kernel is a black box for most people. The code is available,
but it is becoming more complicated and harder to understand, especially
for a new comer. For those that are interested in learning what the
Linux kernel does underneath their systems, ftrace is a very powerful
tool. Ftrace is an internal kernel tracer that was added to mainline
Linux in 2.6.27. It includes a function tracer that allows you to see
what functions the kernel is calling while the kernel is calling them.
There's trace points placed within the kernel that allows the user to
see things like tasks being scheduled in and out, timers triggering,
interrupts taking place, and much more.

This workshop will show how to use ftrace directly, along with trace-cmd
and kernelshark, which are a command line interface as well as a GUI
interface for ftrace. As most distributions now come with ftrace
installed within their kernels, people can just bring their laptops with
Linux and be able to follow along without having to install a new
kernel. Trace-cmd and kernelshark are easy enough to build and install,
that most attendees will be able to use them too.

The focus of the workshop will start out with some simple commands to
get the feel of ftrace. That is, ftrace control files are in the debugfs
filesystem that can be mounted by issuing the mount command
to /sys/kernel/debug. Ftrace is designed to not need any extra tools,
and can be enabled and disabled, as well as reading the trace with just
using the cat and echo commands. More of the advanced features of ftrace
will then be discussed and little tricks that can be done to find things
like the longest running interrupt that causes latencies in your system.

There are user tools to facilitate ftrace, as the control files can
become tedious to use. This includes trace-cmd and kernelshark.
Trace-cmd is a tool that allows you to run various ftrace tracing and it
will record the result to a file (which may even be on another machine).
Then the tool can be used to read the trace as well as manipulate the
output. Having the data in a file gives the user much more control in
parsing and analyzing the trace.

Kernelshark, is a GUI front end to trace-cmd. It shows a graphical
representation of the trace and is also interactive. One can start
traces from the tool as well as filter out various events, or zoom in
and out to see the bigger picture move to a more detailed view.

For more information on ftrace and the above tools, there's some nice
articles by LWN.net:

http://lwn.net/Articles/365835/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/370423/
http://lwn.net/Articles/410200/
http://lwn.net/Articles/425583/