Embedded Software Development

Tracing and Visualization of Embedded Linux Systems

Seite: 2/2

Anbieter zum Thema

Tracing in Linux using LTTng

LTTng is an open source technology for software-based tracing in Linux. LTTng is very efficient, proven in use and supported by most Linux distributions. The LTTng kernel tracer can record scheduling events, system calls, IRQs, memory management, and other kernel-level activities. LTTng user-space tracer (LTTng-UST) tracks user-defined events in application code.

Transparent tracing of function calls using wrapper functions and LD_PRELOAD
Transparent tracing of function calls using wrapper functions and LD_PRELOAD
(Image: Percepio)
Trace data is initially stored in RAM, but can be continuously flushed to disk or offloaded to another system over a network connection. Another option is to keep the trace data in a RAM ring buffer, overwriting earlier events when the buffer becomes full.

Although LTTng is based on software instrumentation, it does not require a rebuild of the Linux kernel. Rather, LTTng leverages tracepoints which already exist at strategic locations throughout the kernel. These tracepoints are placeholder function calls, dormant by default but activated by LTTng.

Using LTTng-UST, developers can trace custom events in user-space application and library code. It is even possible to trace selected function calls without modifying source code, by creating a shared object file with wrapper functions containing tracepoints, specified in LD_PRELOAD when launching the application.

LTTng-UST function wrapping is completely transparent to application code, with no need for recompilation. On the first call, the wrapper function looks up the address of the original function and stores it in a function pointer (as with malloc in the picture above)

Visualization of LTTng traces in Tracealyzer

LTTng outputs binary data files, so a tool is required for analysis. Babeltrace can convert the trace data to text files, but it is hard to “see the big picture” from vast amounts of trace data in text format. A visualization tool greatly facilitates analysis since the human brain is much better at spotting patterns in images than in text data.

Tracealyzer is a family of trace visualization tools developed by Percepio AB in Sweden. Tracealyzer for Linux is designed to visualize LTTng trace data and provide multiple graphical perspectives to facilitate analysis. Tracealyzer understands and highlights dependencies among related events in trace data, for instance sending and receiving of a semaphore signal. Highlighting such dependencies makes it easier to understand operating system behaviour, e.g., why some threads are blocked and others triggered.

The main trace view in Tracealyzer (see the main picture on the previous page) presents events (e.g., system calls) along a vertical time-line using colour-coded labels. Labels can be filtered in several ways and automatically adjust to avoid overlapping. Label background colour indicates status and type of operation, e.g., red labels show system calls that block the calling thread. Custom application events from LTTng-UST can be configured to appear either as service calls (e.g., malloc) or as user events (generic debug messages).

The Tracealyzer main view is complemented by 20+ additional views, showing CPU load, response times, kernel blocking, scheduling intensity, kernel calls and other traced events. User events can be shown separately in a text log view, and user event arguments can be visualized as signal plots. All views are interconnected, meaning that clicking on a task, event or data-point will open another relevant view with the clicked entity highlighted. This interconnection makes it easier to use multiple perspectives when studying a specific interval in the trace.


Tracing provides a powerful tool for analysing multi-threaded software systems. On Linux, tracing is enabled by LTTng, a mature and proven open source solution. Percepio Tracealyzer for Linux lets developers visualize LTTng trace data through multiple, interconnected graphical views.

Tracealyzer makes dense and voluminous trace data more accessible to software developers, giving them greater benefit from tracing. Tracealyzer helps developers make sense of complex trace data, find bugs and tune performance, and thereby produce better software.

* * Johan Kraft is the founder and CEO of the Swedish software company Percepio which specializes in the visualization of tracing results.

Jetzt Newsletter abonnieren

Verpassen Sie nicht unsere besten Inhalte

Mit Klick auf „Newsletter abonnieren“ erkläre ich mich mit der Verarbeitung und Nutzung meiner Daten gemäß Einwilligungserklärung (bitte aufklappen für Details) einverstanden und akzeptiere die Nutzungsbedingungen. Weitere Informationen finde ich in unserer Datenschutzerklärung.

Aufklappen für Details zu Ihrer Einwilligung