The trivial approach of setting a breakpoint sure didn't work. Somehow in all the forking around, Gdb failed to remove the break point in the child. So, I got the amusing message
Program terminated with signal SIGTRAP, Trace/breakpoint trap. The program no longer exists.With that auspicious start, I began my adventure. I'll skip the play-by-play , but I want to pass along some useful observations.
- The best way to deal with forking is to run the program, find the right process through some other means and attach to it. Life is very frustrating when this doesn't work because you need an early breakpoint, because it's hard to find the process or the like.
- When that fails, the catch fork command can be used to break after a fork succeeds.
- Don't forget to set follow-fork to indicate whether the debugger should stay with the parent or child. (I really wanted the semi-mythical set follow-fork both)
- I found the amazing set scheduler-lock command. This allows you to disable execution of other threads while you're debugging.
- Don't forget to turn off the scheduler lock from time to time: multi-threaded programs get into some fairly unusual deadlocks when only one of the threads is permitted to run.