Debugging multithread apps is not easy task. Some problems never occur in debugging mode and it could be discovered only when application is running for long time. One of problematic issues in multithread apps are mutexes.

There is very simple tool in Debian distribution that could help you with identifying problems mutexes. The tools is mutextrace.

Quote from package description:

This tool displays mutexes locked and unlocked during execution of a program, and can provide hints for debugging multithreaded applications. As it is fairly lightweight and easy to use, it is ideal at the start of a debugging session or when only limited resources are available.

It cannot detect data races as it only looks at the synchronization primitives themselves; if you need that, use helgrind (in the valgrind package) instead.

How to use mutextrace? Like strace. Just add mutextrace before your application in the shell:

mutextrace ./mutex-tester-app param1 param2

Result:

lock #1, free -> thread 1
init #2
lock #2, free -> thread 1
unlock #1
lock #1, free -> thread 1
init #3
lock #3, free -> thread 1
unlock #1
unlock #3

Simple :-)

I couldn’t find reference to original source of mutextrace. It seems that it does not exist.