Root > Solving bugs in your code > Bug reports > Call Stack section > How to read call stacks > Multi-threaded call stacks

Multi-threaded call stacks

Previous pageReturn to chapter overviewNext page   

This article is a part of working with bug reports.

 

Important note: please read Using EurekaLog in multi-threaded applications article before reading this article.

 

Call stacks in multi-threaded applications contain additional information.

 

For example:

 

 

Exception in background thread (GUI)

 

Call Stack Information:

----------------------------------------------------------------------------------------------

|Address |Module      |Unit             |Class       |Procedure/Method            |Line      |

----------------------------------------------------------------------------------------------

|*Exception Thread: ID=2940; Parent=3696; Priority=0                                         |

|Class=TMyThread; Name= (Unit1.TMyThread.Execute)                                            |

|DeadLock=0; Wait Chain=                                                                     |

|Comment=                                                                                    |

|--------------------------------------------------------------------------------------------|

|0069209C|Project1.exe|Unit1            |            |DoCrash                     |77[0]     |

|006920CA|Project1.exe|Unit1            |TMyThread   |DoWork                      |82[2]     |

|00692080|Project1.exe|Unit1            |TMyThread   |Execute                     |69[1]     |

|004DCA96|Project1.exe|System.Classes   |            |ThreadProc                  |14569[12] |

|004094AC|Project1.exe|System           |            |ThreadWrapper               |21627[45] |

|005919AE|Project1.exe|EExceptionManager|            |DefaultThreadHandleException|2852[5]   |

|0056F32B|Project1.exe|EThreadsManager  |            |ThreadWrapper               |611[11]   |

|75B9F26F|kernel32.dll|kernel32         |            |BaseThreadInitThunk         |          |

|00692075|Project1.exe|Unit1            |TForm1      |Button2Click                |63[2]     |

|--------------------------------------------------------------------------------------------|

|Calling Thread: ID=3696; Parent=0; Priority=0                                               |

|Class=; Name=MAIN                                                                           |

|DeadLock=0; Wait Chain=thread: [ 0E70 / 3696 ] is blocked                                   |

|Comment=                                                                                    |

|--------------------------------------------------------------------------------------------|

|76718390|USER32.dll  |USER32           |            |NtUserWaitMessage           |          |

|006811AB|Project1.exe|Vcl.Forms        |TApplication|HandleMessage               |10238[1]  |

|006814D9|Project1.exe|Vcl.Forms        |TApplication|Run                         |10376[26] |

|0069BCFD|Project1.exe|Project1         |            |Initialization              |22[4]     |

|75B9F26F|kernel32.dll|kernel32         |            |BaseThreadInitThunk         |          |

----------------------------------------------------------------------------------------------

 

Exception in background thread (text)

 

It contains 2 call stacks: one per each thread. There are two threads in this application: main thread and a background thread based on TThread class. There was exception raised in the background thread.

 

This example illustrates several multi-threaded features of EurekaLog:

 

 

Call stacks for multiple threads

Bug report contains a single call stack by default. This call stack is created for thread which raised the exception. Such thread is called "Exception thread". This is the only thread in single-threaded application (it will be main thread for single-threaded application). Sometimes you may be interested in other threads, you may want to know what other threads are doing when exception had occurred. You can capture call stacks of other threads in application by checking "Capture stack of RTL threads" or "Capture stack of Windows threads" options.

 

Notes:

there may be multiple call stacks for leak reports. There will be one call stack per each found leak. This behavior is customizable here. This behavior is not related to multi-threading;
capturing call stack of an external thread requires thread's suspending. In rare case this can cause deadlock issues (for example: thread may be suspended when it is running memory allocation function; thus, any further memory alloc/release operation will block application forever). Do not enable this option until really needed.

 

 

Different call stacks types

EurekaLog marks call stacks as:

Exception thread. This is thread which raised the exception. There can be only one exception thread in any bug report. This thread is listed first. It is mandatory; any other threads are optional;
Calling thread. This is parent thread for exception thread. Normally, there is only one calling thread in bug report. There may be more than one calling thread if exception occurred inside synchronized routine (see below). This thread follows exception thread in the list. It's optional. It will be not shown if this thread was terminated before exception or if the corresponding "Capture stack of XYZ threads" option is turned off.
Running thread. Any other thread is called "running thread". Main thread can also be a running thread. All running threads are listed after exception thread and calling thread(s) (if any).

 

Note: the meaning of exception and calling threads are changed significantly for exceptions within synchronized methods (see below).

 

 

Different thread types

EurekaLog can recognize the following thread types:

RTL theads. These are threads created by Delphi/C++ Builder code. Such threads are created via:
oTThread or TThreadEx class;
Windows threads. Any other threads are considered as "Windows threads". Such threads are created via CreateThread function.

 

It's recommended to create your own threads via TThreadEx class or via BeginThreadEx function (or via any other function which use TThread or BeginThread to create threads). Do not use CreateThread function to create threads. This will allow you to distinguish between your threads and system threads (system or 3rd party code may create additional threads in your application for background/service tasks).

 

If you're using TThread(Ex) approach - then EurekaLog will be able to extract (child) class name and show it in the call stack's header as "Class=TMyThread".
If you're using BeginThread(Ex) or CreateThread approach - then EurekaLog will be able to extract thread function and show it in the call stack's header as "Name= (ThreadProc)". Thread function's name will be printed in parentheses (see "Naming threads" below to know more about "Name" parameter). Thread function's name will always be Execute method for TThread approach.

 

 

Naming threads

You can name threads to simplify debugging. Since you can run multiple threads with the same thread class (or thread function) - this means that you can not distinguish between these threads by using only class name (or thread function's name). You need some additional marker. Such marker is a thread name.

 

You can supply thread name directly to TThreadEx class or BeginThreadEx function:

 

uses
  EBase;
 
  TH := BeginThreadEx(nil, 0, MyThreadProc, Args, 0, TID, 'This is my thread');
 
// or:
  
type
  TMyThread = class(TThreadEx)
  // ...
  protected
    procedure Execute;
  // ...
  end;
 
procedure TMyThread.Execute;
begin
  inherited;
  // ...
end;
 
  Thread := TMyThread.Create(False, 'This is my thread');

 

If you are not using TThreadEx/BeginThreadEx - then you can name a thread from the thread itself. You can set thread's name by using NameThread function from EThreadsManager unit. You should call this function as a very first action inside thread function (for BeginThread) or Execute method (for TThread). Thread name can be arbitrary string, so you can append any parameters that you need to distinguish between threads. For example:

 

{$IFDEF EUREKALOG}

uses
  EThreadsManager;

{$ENDIF}
 
type
  PThreadArguments = ^TThreadArguments;
  TThreadArguments = record
    FileName: String;

    // ... any other arguments
  end;
 
function MyThreadProc(Parameter: Pointer): Integer;
var
  Args: TThreadArguments;
begin
  Args := PThreadArguments(Parameter)^;
  Dispose(PThreadArguments(Parameter));
 
  {$IFDEF EUREKALOG}NameThread('FileName=' + Args.FileName);{$ENDIF}
 
  // ... actual thread's code goes there
end;
 
...
 
var
  Args: PThreadArguments;
  TID: Cardinal;
  TH: THandle;
begin
  Args := AllocMem(SizeOf(TThreadArguments));

 
  Args.FileName := Edit1.Text;

 
  TH := BeginThread(nil, 0, MyThreadProc, Args, 0, TID);

 
  if TH = 0 then
    Dispose(Args)
  else
    CloseHandle(TH);
end;

 

Any thread name will be shown in the call stack's header as "Name=your-thread-name" (without parentheses; part inside parentheses is a name of thread function - see above). For the example (the one directly above) it will be:

 

Name=FileName=C:\Sample.txt (MyThreadProc)

 

(assuming that Edit1.Text holds 'C:\Sample.txt' string)

 

Note: main thread will always have a name "MAIN".

 

 

Indicating creator of the thread

Since you can run multiple threads with the same thread class (or thread function) and with the same arguments from multiple locations - this means that you can not distinguish between these threads by using class name (or thread function's name) and any properties of the thread itself. You need to use name of thread's creator.

 

EurekaLog appends creator of the thread to the end of thread's call stack. Typically this is a single entry below BaseThreadInitThunk system function. For example, this example of the call stack shows that TMyThread was created in Unit1.TForm1.Button2Click method, line 63:

 

Call Stack Information:

-----------------------------------------------------------------

|Address |Module      |Unit    |Class |Procedure/Method   |Line |

-----------------------------------------------------------------

|*Exception Thread: ID=2940; Parent=3696; Priority=0            |

|Class=TMyThread; Name= (Unit1.TMyThread.Execute)               |

|DeadLock=0; Wait Chain=                                        |

|Comment=                                                       |

|---------------------------------------------------------------|

|0069209C|Project1.exe|Unit1   |      |DoCrash            |77[0]|

<- ..........................................................->

|75B9F26F|kernel32.dll|kernel32|      |BaseThreadInitThunk|     |

|00692075|Project1.exe|Unit1   |TForm1|Button2Click       |63[2]|

|---------------------------------------------------------------|

 

 

Notes:

call stack of the creator thread is not captured by EurekaLog, only single entry (immediate caller) is captured;
call stack of the calling thread is not the call stack for the creator. Do not confuse these call stacks. Calling thread indicate state of creator's thread after thread creation had occurred;
do not confuse last line in thread's call stack ("creator") as belonging to the thread itself. This code was executed by different thread (indicated by "Parent"). This line was not executed by the thread itself.

 

 

Indicating threads options

EurekaLog lists some options of threads, such as:

ID value contains TID (Thread ID). This is system value that unique identifies threads within the system. Do not confuse TID with thread's handle - those are totally different things. TID (together with PTID - see below) can be used to check relations between threads;
Parent value contain PTID (Parent Thread ID). This is ID of thread which created this thread. This value is always 0 for the main thread. Please note that thread identified by PTID may be already finished and terminated when exception had occurred;
Priority value indicate thread's priority as returned by GetThreadPriority function.

 

 

Wait Chain Traversal feature support

EurekaLog has support for WCT (Wait Chain Traversal) feature available since Windows Vista. WCT is a mechanism for debugging blocked threads and processes, and detecting deadlocks.

 

Using WCT, debugging software can analyze executing processes and report on the state of threads, including information such as what a blocked thread is waiting for and whether a deadlock condition exists. Debugging software can analyze and then reports the state of threads. A thread's state (unblocked, blocked, or deadlocked) is reported as a wait chain. A wait chain is an alternating directed graph of threads and synchronization objects. In the graph, an edge from a thread to an object indicates that the thread is waiting for the object; an edge from an object to a thread indicates that the thread is the current owner of the object. For example, the following wait chain represents a thread (Thread A) that is blocked waiting for a mutex object that is owned by Thread B.

 

Thread A -> Mutex 1 -> Thread B

 

As illustrated here, the first node in a wait chain is the thread being analyzed and the remaining nodes are the elements (synchronization objects, processes, threads, and so on), if any, that are directly or indirectly causing the thread to block. The simplest wait chain reflects a thread that is not blocked. It is composed of a single node, representing the unblocked thread. A blocked thread is represented by a wait chain containing multiple nodes that is non-cyclic: that is, there are no two nodes in the chain that represent the same thread or object. When a thread is deadlocked, the wait chain that represents it is cyclic. Consider the scenario where Thread A owns Object 1 and is blocked waiting for Object 2, while Thread B owns Object 2 and is blocked waiting for Object 1. The wait chain is as follows:

 

Thread A -> Object 2 -> Thread B -> Object 1 -> Thread A

 

Note that Thread A appears twice. The second node for Thread A could be replaced by an edge (->) that connects Object 1 to the first Thread A node, creating a loop that represents the cyclic dependency.

 

EurekaLog shows 1 in "Deadlock" value if deadlock (cyclic dependency) was detected for this thread. Otherwise "Deadlock" value is 0;
EurekaLog shows thread's wait chain in "Wait chain" value. Simplest wait chains (that is wait chain which consists of a single node) are not shown ("Wait chain" value will be empty).

 

WCT feature is accessible only on Windows Vista and above. "Deadlock" and "Wait chain" values will be empty on older OS versions.

 

 

Parenting threads

EurekaLog lists TID ("ID") and PTID ("Parent") values to help you identify child/parent threads. Additionally, a parent thread for exception thread is marked as "Calling thread" and is shown below exception thread for your convenience. It will be not shown if this thread was terminated before exception or if the corresponding "Capture stack of XYZ threads" option is turned off.

 

 

Merging call stacks for Synchronize

EurekaLog will merge call stacks of main thread and invoking thread (that is the thread which called Synchronize method) when possible. The result may look like this:

 

 

Exception inside synchronized routine (GUI)

 

Call Stack Information:

--------------------------------------------------------------------------------------------------

|Address |Module      |Unit             |Class           |Procedure/Method            |Line      |

--------------------------------------------------------------------------------------------------

|*Exception Thread: ID=400; Parent=0; Priority=0                                                 |

|Class=; Name=MAIN                                                                               |

|DeadLock=0; Wait Chain=thread: [ 0190 / 400 ] is blocked                                        |

|Comment=                                                                                        |

|------------------------------------------------------------------------------------------------|

|0069211C|Project1.exe|Unit1            |                |DoCrash                     |81[1]     |

|0069214F|Project1.exe|Unit1            |TMyThread       |DoWork                      |85[1]     |

|004DC958|Project1.exe|System.Classes   |                |CheckSynchronize            |14525[30] |

|------------------------------------------------------------------------------------------------|

|Calling Thread: ID=2072; Parent=400; Priority=0                                                 |

|Class=TMyThread; Name= (Unit1.TMyThread.Execute)                                                |

|DeadLock=0; Wait Chain=                                                                         |

|Comment=                                                                                        |

|------------------------------------------------------------------------------------------------|

|004DD5A2|Project1.exe|System.Classes   |TThread         |Synchronize                 |15085[38] |

|006920B8|Project1.exe|Unit1            |TMyThread       |Execute                     |71[3]     |

|004DCA96|Project1.exe|System.Classes   |                |ThreadProc                  |14569[12] |

|004094AC|Project1.exe|System           |                |ThreadWrapper               |21627[45] |

|005919BE|Project1.exe|EExceptionManager|                |DefaultThreadHandleException|2853[5]   |

|0056F32B|Project1.exe|EThreadsManager  |                |ThreadWrapper               |611[11]   |

|75B9F26F|kernel32.dll|kernel32         |                |BaseThreadInitThunk         |          |

|00692085|Project1.exe|Unit1            |TForm1          |Button2Click                |63[2]     |

|------------------------------------------------------------------------------------------------|

|Calling Thread: ID=400; Parent=0; Priority=0                                                    |

|Class=; Name=MAIN                                                                               |

|DeadLock=0; Wait Chain=thread: [ 0190 / 400 ] is blocked                                        |

|Comment=                                                                                        |

|------------------------------------------------------------------------------------------------|

|76718390|USER32.dll  |USER32           |                |NtUserWaitMessage           |          |

|006811AB|Project1.exe|Vcl.Forms        |TApplication    |HandleMessage               |10238[1]  |

|006814D9|Project1.exe|Vcl.Forms        |TApplication    |Run                         |10376[26] |

|0069BCFD|Project1.exe|Project1         |                |Initialization              |22[4]     |

|75B9F26F|kernel32.dll|kernel32         |                |BaseThreadInitThunk         |          |

--------------------------------------------------------------------------------------------------

 

Same exception in text

 

The above example should be translated as:

there are two threads in application: main thread (ID = 400) and background/worker thread (ID = 2072). Although there are 3 call stacks in the report, but there are only two unique threads (as indicated by IDs). There are two call stacks for main thread (ID = 400): first call stack is exception thread and it was created inside Synchronize call, second call stack is usual call stack for main thread;
worker thread (ID = 2072) was launched from Unit1.TForm1.Button2Click line 63[2]. This is indicated by last line in call stack for thread #2072 - right below system's BaseThreadInitThunk. Main thread does not have creator - this is indicated by BaseThreadInitThunk being the last in the second call stack for the main thread (ID = 400);
worker thread (ID = 2072) called Synchronize method. This is indicated by presence of TThread.Synchronize at the top of the call stack for thread #2072;
task from Synchronize method was executed by main thread (ID = 400). Task is DoWork method from TMyThead class. This is indicated by TThread.DoWork above CheckSynchronize call for main thread (ID = 400);
there was exception raised in DoCrash routine which was called from TMyThread.DoWork. This is indicated by exception call stack - which is the first call stack for main thread (ID = 400);
main thread waits for window message (i.e. it is processing messages; it's inside message loop). This is indicated by last call stack for main thread (ID = 400).

 

Notes:

First call stack for main thread ("exception thread") does not represent any real-time information. Main thread is doing message pumping at the current moment - as indicated by the second call stack for main thread. Exception thread shows some older state of main thread - that is the state when main thread was doing tasks for Synchronize. In other words, there are two call stacks for the single thread (main thread): one represents the past and second represents the current moment.
"Exception thread" is not a real exception thread. The handled exception was (re-)raised by TThread.Synchronize wrapper which was called from the background thread. This means that exception thread is the background thread, not the main thread (as call stack suggests). However, main thread is still considered as exception thread.
Call stacks for manual synchronization (such as SendMessage into main thread) will not be merged.
This feature supports merging call stacks for any exceptions that was re-raised into another thread. Synchronize calls is just a sub-case of generic cross-thread exceptions.
This feature supports any TThread-descendant - such as AsyncCalls or OmniThreadLibrary.

 

 

See also:




Send feedback... Build date: 2018-11-26
Last edited: 2018-06-14
PRIVACY STATEMENT
The documentation team uses the feedback submitted to improve the EurekaLog documentation. We do not use your e-mail address for any other purpose. We will remove your e-mail address from our system after the issue you are reporting has been resolved. While we are working to resolve this issue, we may send you an e-mail message to request more information about your feedback. After the issues have been addressed, we may send you an email message to let you know that your feedback has been addressed.


Permanent link to this article: https://www.eurekalog.com/help/eurekalog/multithreaded_call_stacks.php