Brendan Gregg's Blog

I gave this demo recently at USENIX/LISA 2016, showing ftrace, perf, and bcc/BPF. A video is on youtube:

It was part of a larger talk on Linux 4.x Tracing Tools using BPF Superpowers. The slides are on slideshare:

Update: I've now posted about the full talk, which is also on the usenix website. Apologies for the audio: I had some typing sound effects that sounded soft on stage, but came out too loud in the recording.

My 15 (18) minute demo stepped through the evolution of recent built in Linux tracers: ftrace (2008+) and its many capabilities, perf (2009+), and bcc/BPF (2015+) which provides the final programmatic abilities for advanced tracing, via enhanced BPF (aka eBPF). I suspect I might change people's view of Linux tracing, as these tracers – despite being built in to the Linux kernel – are still not widely known.

perf & BPF tutorial at LISA 2016

Earlier at the conference, Sasha Goldshtein and I ran a half day perf & bcc/BPF tutorial. Both Sasha and I are not only bcc contributors, but also experienced classroom instructors, and it was a pleasure to collaborate with him on this project. It wasn't videoed, but the lab files are on github. If you are interested in learning bcc/BPF, there are also two tutorials I wrote in bcc/docs for using and developing bcc tools.

There was a lot of interest in both our tutorial and my talk – I imagine this interest will grow over time as more people deploy on Linux 4.x series kernels and can make use of BPF.

For more about Linux tracers, here are some resources:

• ftrace: The hidden light switch (lwn.net), perf-tools (github), ftrace.txt
• perf: perf Examples, perf wiki
• bcc: bcc/BPF tools, and many posts here (listed on Linux Performance).

Then there's also the add on tracers, like Systemtap, LTTng, sysdig, etc, which I didn't cover in 15 minutes.

My 15 minute tracing demo was inspired by Greg Law's excellent cppcon talk Give me 15 minutes & I'll change your view of GDB. Since then, I've also written about GDB here, with a full GDB example (tutorial).

LISA was a lot of fun. Thanks to those who were able to attend our events, and USENIX for putting on another great conference!