Windows 10: How to track down that irritating bug with DTrace

Microsoft has implemented the popular open tracing tool to help developers debug Windows code.

Microsoft Ignite 2019 in review: Azure Arc, HoloLens 2, Edge, Quantum, and Teams
3:43

If you've been using Unix or Linux to build complex applications, you should be familiar with DTrace. Originally developed by Sun, it's a dynamic tracing tool that allows you to drill down into much of what's happening in a running system, from the hardware, up through the OS, to your applications. This powerful tool gives you a lot of information that can help debug and tune your code.

What is DTrace?

Using the command line and its own scripting language, D, you can write tracing programs, assigning actions to instrumentation probes. You can think of a DTrace probe as a multimeter lead connected into your OS or your code, waiting for a certain condition to be met and then logging the appropriate information. One of the benefits of DTrace is the ability to use more than one probe, providing the tools you need to understand how events are related, and helping to track down complex bugs that traditional debugging tools can't pinpoint.

Where many debugging tools will have an effect on your code's performance, DTrace has been designed to have as little impact as possible. That's an important difference, and one that allows you to have hundreds or even thousands of probes. It's a model that works well with modern distributed applications, as it shows how microservices are operating and gives you observability that you wouldn't have had before.

DTrace's D language is designed to be like C, so it's not that hard to learn. The same syntax used in scripts is used in command-line calls, which provide a quick way of putting probes into specific operations, showing the results on-screen. The command line can help with testing and debugging scripts; you can try probes out interactively before adding them to a more complex set, understanding the format of the outputs that can be used to drive other probes.

Microsoft's implementation

Microsoft has now ported DTrace to Windows, building on the Open DTrace code and specification, adding specific Windows features with support for Event Tracing for Windows (ETW), for Windows system calls, and for Windows Process IDs. While Windows has a very different architectural model from Unix and Unix-derived operating systems, you should find DTrace works well with Windows, especially if you're using the Windows-specific tracing features.

While Windows has its own ETW tracing tools, they're static and need to be built into your code. DTrace is dynamic and has the advantage of being able to work at runtime, letting you quickly instrument and trace running code. It's even possible to quickly instrument a production system, get the details needed to debug a problem, removing the DTrace tooling and configuration when you're done.

dtrace-on-windows.jpg

Users interact with DTrace via the dtrace command, a generic front-end to the DTrace engine. D scripts are sent to the DTrace kernel component for execution, sometimes called as the DIF Virtual Machine. Traceext.sys is a new kernel extension driver added by Microsoft, which allows Windows to expose functionality that DTrace relies on to provide tracing.

Image: Microsoft

Getting started with DTrace

Installing DTrace isn't as easy as installing a traditional Windows application. It's a very low-level tool, requiring a couple of reboots to get running. First download the appropriate version of DTrace from Microsoft. The current supported version only runs on Windows 10 2004 or later; an earlier archived version for 1904 and 1909 can still be downloaded, but it's no longer supported.

Once downloaded, install the DTrace application and then use BCDEdit to add it to your Windows boot configuration. You may need to turn BitLocker off if you're using it to protect your PC's boot configuration before you do this, otherwise you may need to enter your BitLocker key when you next boot.

Next, set your Windows PATH environment variable to reference DTrace, as well as adding a new environment variable for any debugging symbols. These will be downloaded from Microsoft when they're needed. Finally, make sure that you've set up Virtual Secure Mode to trace on the Windows kernel boundary.

SEE: 20 pro tips to make Windows 10 work the way you want (free PDF) (TechRepublic)

Many developers test on virtual machines, and DTrace will run inside a VM as long as it supports nested virtualisation. You can do this from PowerShell, enabling virtualisation extensions and then rebooting the host PC.

One important caveat is you do need to be running as a local admin account to get DTrace working properly, so if you're using a Microsoft Account to log into Windows 10, be ready to add a new local user. Remember to re-enable DTrace via BCDEdit when you update Windows Insider builds.

Debugging with DTrace

With DTrace ready to run, you can use command-line single line tools to see available probes and try out simple queries, getting to grips with the structure of D commands. Once you're familiar with these, you can start building your own scripts.
 
Microsoft provides a section of sample scripts on its DTrace GitHub page, which can help you get started with tracing across your applications. It's worth sending some time with a book on D, as programming trace scripts can be complex, especially as probes are likely to fire asynchronously. One good place to start is the free Dynamic Tracing Guide book on the dtrace.org website. While it's technically for illumos (an open-source OS based on Open Solaris), the concepts and processes in it can be translated to Windows.

The sample scripts are split into groups, looking at common Windows tracing scenarios. You'll find some for looking at disk operations, others for files, for memory, and for use with ETW. Download the code and run it on your systems, before making your own versions targeting your specific needs.

Results can be hard to analyse, especially when you're presented with files full of numbers. It's a good idea to use tools like Excel or PowerBI to build visualisations and explore trace data to help understand DTrace results.

DTrace is a powerful tool, and it's good to see Microsoft bringing it to Windows. But it's clearly not for beginners, with a relatively complex install and a steep learning curve. Once you've mastered its D language and learned how to choose and implement probes, then you're ready to explore how your applications interact with each other and with Windows. If you're trying to track down a tricky bug, this is the tool for you, opening a whole new window into Windows.

Additional resources