Embedded Software Development Tracing and Visualization of Embedded Linux Systems

Autor / Redakteur: Johan Kraft * / Franz Graser

Tracing is an important method to detect program errors in embedded systems. Visualization tools lend support to a better understanding of the tracing results.

Anbieter zum Thema

Tracealyzer main view - scheduling and various events.
Tracealyzer main view - scheduling and various events.
(Image: Percepio)

Debugging embedded software can be a challenging, time-consuming and unpredictable factor in development of embedded systems. Detecting errant program execution begs the question “How did the software reach this state?” What combination of inputs and timing resulted in the error, and why?

Tracing can often provide the answer.

Tracing entails recording software behaviour during runtime, allowing for later analysis of collected trace data. Tracing is most often a development bench activity, but tracing can also be enabled for production use, continuously active to record behaviours and catch errors post-deployment. Production tracing can be an effective technique for detecting rarely-manifested errors that are therefore are difficult to reproduce in a debugger. These can include situations where the system responds more slowly than expected, gives incorrect or suboptimal output, freezes up or crashes.

Tracing can be performed either in hardware (in the processor) or in software. Hardware-based tracing generates a detailed instruction-level execution history, while software-based tracing focuses on selected events, typically in the operating system and important application-level interfaces.

Hardware-generated trace provides details regarding control-flow and does not impact the execution of the traced system, but does require special equipment and a trace-enabled hardware platform.

Software-generated trace does not require any special hardware and can even be deployed in shipped products like a “black box” flight recorder used in aviation. Moreover, software trace allows for storing data, such as system call parameters, while hardware trace is typically limited to control-flow only. Software tracing does induce some CPU overhead, but typically less than 1 %. Software tracing relies on target system RAM for storing the trace data, at least initially, but the RAM buffers are usually configurable to allow for balancing RAM usage vs. execution history length.

Tracing is especially important for systems integrating an operating system. A central feature of OSes is multi-threading - the ability to run multiple programs (threads) on a single processor core by rapidly switching amongst execution contexts. Multi-threading, however, makes software behaviour more complex, and affords the developer less control over run-time behaviour as execution is pre-empted by the OS.