EurekaLog
Menu

The EurekaLog Developer Guide

27. Call stacks that did not resolve

A useful call stack needs both a plausible chain of addresses and a way to describe those addresses. Missing frames, missing symbols and filtered entries are different failures. Start by asking which one your report shows; changing symbol providers cannot repair execution that jumped into invalid memory, and changing the tracer cannot manufacture a package's missing line information.

Chapter 11 explains the tracing methods and debug-information providers, chapter 20 explains how to read their output, and chapter 2 covers repeatable builds. The cases here show the discriminators that separate a configuration repair from a toolchain limit or a product defect.

27.1 How the cases are labelled

A customer observation is what the reporter saw; a staff proposal is advice given but not necessarily confirmed; a staff diagnosis is support's explanation of the cause. An explicit confirmation is the reporter agreeing that it matched, and an independently reproduced result is the vendor reproducing it in a project of its own. Closed is not fixed: several cases below end with "as expected", which is a statement about the product at that version, not every release.

27.2 Where a name comes from, and the two ways it can fail

The lookup starts from an address and asks which module contains it; the module selects the debug information source, and the source answers with a record - module, unit, class, routine, line, and the file the debug information came from (chapter 11, sections 11.6 and 11.7). Names can therefore fail in two different places, and the reports look different:

  • nothing was built: the module carries no debug information at all, so the source has nothing to answer with - the documented detail level for that outcome says "code belongs to module, but module does not have debug information, or code lies outside of debug information scope";
  • something was built but could not be read: format, provider, deployment or encryption - the answer is missing although the information exists somewhere.

Inspect the built image before changing your code. EurekaLog's executable analysis identifies whether the module was post-processed and which debug information it carries. Compare that result with the actual deployed files, including packages and external symbols; see chapter 4 for the tools workflow.

27.3 Names that were never built

A release profile with debug information switched off. With EurekaLog 7.13.3.0 Update 3, Delphi 12 Athens and Win32, reports reached the issue tracker but the viewer showed no source line numbers. Support suggested the Release target's debug settings and unit-level directives. The customer confirmed that Debug Information was set to No Debug Information in Release. This confirms the missing input; it does not independently establish that every other pipeline setting was correct.

The build must produce the raw material. EurekaLog requires the linker's map file: it uses it to compile the debug information it injects, so the linking options have to generate one, which the tool normally does by itself (staff diagnosis). The second half is easy to miss and is documented: the map contains only what the precompiled files expose, so the compiler's own debug information must also be enabled, otherwise information about the source does not reach the map at all. Using the debug versions of the runtime's own units is what adds line numbers for those units (staff diagnosis). A package project makes the same point from the other side: with the project's debug setting off, a lot of the important information disappears from the map, and unit names, method names and line numbers go with it (staff diagnosis). In the same family, a compiler directive that turns debug information off inside a source file or an included file removes the unit from the map even when the project is configured correctly (staff proposal: check for the directive, and place its positive form at the top of the unit).

The reader has to be in the build. A deliberately minimal setup using EurekaLog 7.9.5.0, Delphi 10.4 and Win64 produced addresses without names even though executable analysis found EurekaLog data. Support identified the missing EDebug* reader units: EDebugEL reads injected EurekaLog information, EDebugMap reads MAP files and EDebugExports reads exported names. Include the reader for the format actually built and deployed (chapter 11).

The customer obtained readable stacks with EDebugEL and retained that solution. The JEDI route remained unsuccessful in the customer's setup, although support tested EDebugJCL with JCL 2.8.0.7718 successfully for executable frames. That does not prove either a universal JEDI failure or that the two configurations were equivalent.

The symbols were moved outside the executable. The options that store debug information outside the binary exist for size, deployment or secrecy, and they can fail in a way that looks like a broken stack: a developer who stored all names externally got reports without unit or class names, in both a Delphi and a C++ project, including a report produced on a second machine and opened on the development one, while the options that keep the names produced correct stacks (customer observation). Support's reading was that the application could not find or load the external symbols, that this should not happen, and that a run-time log was needed to see whether it was a defect (staff proposal). That is where the evidence stops: "no names" may be a build setting, a file missing next to the executable, or a genuine defect.

Encrypted or compressed names. A report opened in a text editor showed gibberish where the names of the application's own module should have been, while the system libraries beside them were readable: an encryption key scrambles the appended debug information, and the report has to be opened with a viewer given the same key (staff diagnosis). Later symbol tables are stored as a separate symbol file whose location must be known to the reader (staff diagnosis). The first case is from the older major version, and its answer notes that the viewer version had to match.

27.4 Platform and toolchain limits

C++ Builder and line numbers. A C++ project whose exception reports carried routine names but no line numbers, in a 32-bit build with IDE 10.3 Rio, produced a diagnosis that is worth memorising because it covers three configurations (staff diagnosis, documented): a C++ Builder map file contains no line numbers at all; for 32-bit builds the tool restores them from the Turbo Debugger format, which is why that information has to be enabled in the linker options; and a 64-bit C++ build does not use that format at all - it is built on a toolchain whose debug format the product could not read at the time. The same answer names the case that triggers the most confusion: using the newer compiler for a 32-bit build takes the Turbo Debugger information away as well, so line numbers go with it. A separate case shows the same limitation in a 64-bit build with IDE 11 Alexandria and adds that the IDE debugger's own behaviour differs between the two toolchains, including the debugger's habit of failing to load debug information when an executable's timestamp changes - which is why the tool saves and restores file times when it modifies an executable (staff diagnosis).

The limit is line numbers, not the product. A developer read the documented note about the 64-bit C++ toolchain as "EurekaLog cannot be used for 64-bit C++ projects" and asked about the contradiction with the product's public claims. Support's answer separates the two facts: the note concerns line numbers only, exception reporting could still be used. The actual problem in that case was different: the executable had not been post-processed at all, which image analysis had already revealed. Keep an unsupported line-number format separate from an entirely missing EurekaLog build step.

27.5 The module the frames belong to

A frame's name comes from the module that contains its address, so a stack is only as complete as the binaries it passes through. The recurring shape is a stack whose top or middle belongs to a binary somebody else built.

  • A package supplied without debug information. In EurekaLog 7.11.1.10 Beta 10, Delphi 10 Seattle, Win32, support read a report as showing a package without symbols and an application unit without line information. The proposed checklist was concrete: enable DLL Exports for exported names; use JEDI for a matching .jdbg, Synopse for .mab, TD32 for .tds, or the MAP provider for .map. A provider only helps if its input exists and matches the package. Otherwise obtain a symbol-enabled package from its author or rebuild it yourself. For your own units, check compiler debug settings, debug DCUs and {$D-} directives in both source and include files. This exchange has no customer retest; it is a diagnosis and checklist, not a demonstrated configuration fix.

  • Entries dropped rather than unnamed. With EurekaLog 7.12.0.4 Hotfix 4, Delphi 11 Alexandria and Win32, the viewer showed a one-line stack. Support identified a third-party database package without debug information combined with a detail level admitting only entries with line numbers. It proposed allowing procedure-name entries, deploying package symbols and enabling the matching provider. The same conversation included other reports, invalid execution addresses and a possible recursive failure in exception collection; it was not a clean one-setting experiment.

    The final customer message reported a full stack with 7.12.0.5 Hotfix 5. It did not establish that changing detail level alone had restored it, nor identify which change in the build mattered. Keep the useful filter/symbol checklist and the positive upgrade retest, but do not confuse them.

  • A name that is a guess. Without debug information, an export-table provider names the frame after the nearest exported symbol, so an entry can read as a library routine with a large offset from it - a real name, but not the routine that ran (staff diagnosis); those providers are documented as heuristic, so a third-party frame is best read as "somewhere in this library" until that library ships symbols.

27.6 The tracing method decides which calls exist at all

Two families of tracing method exist, and chapter 20, section 20.3 explains how they differ; the point that matters here is that they do not fail in the same place.

Frame-based tracing can only see calls that have a frame. The runtime and the framework are compiled with the frame option turned off - and you do not recompile them - so short routines there are invisible to this method and the stack is incomplete through them (staff diagnosis, documented). The consequence is described with one example: a leaked object created inside a click handler will not produce a stack naming that click handler, because the constructor that ran is a short runtime routine without a frame, and a frame identifies the caller rather than the routine it belongs to (staff diagnosis). The same answer notes that calls through the runtime's standard memory routines are identified anyway, because the tool knows those routines and extracts their caller whatever the tracing method - which is why some leaks are named and similar ones are not.

RAW tracing sees more, including things that are not there. It scans the whole stack guessing which values look like return addresses, so it recovers calls the frame method cannot see and at the same time produces entries from stale stack contents; the documentation states that false-positive entries cannot be removed with full efficiency and that a RAW stack has to be analysed (documented, chapter 20, section 20.3).

The choice interacts with leaks in the opposite direction. For memory reports a frame-based method cannot build a stack for blocks allocated by code without frames, and a leak whose stack is empty is not worth showing, so those leaks are hidden - leaving only "there is a leak of some bytes" - while RAW builds a stack for each block and therefore shows more blocks (staff diagnosis). The recommendation is consistent and documented: RAW for exceptions, which are rare, and the cheaper frame method for leaks, which follow every memory operation; never deploy a build with RAW leak tracing enabled (chapter 22, section 22.7). The same case is a reminder that "RAW shows more leaks" can also mean "RAW shows something that is not a leak you own", which is where chapter 26 continues.

27.7 Stacks that came back empty, or nearly empty

  • No debug information anywhere near the return addresses. A developer's own routine built a stack for a running thread and got one line, an address in a system library. The analysis is that the addresses found on the stack have no debug information behind them, with the recommendation to build a debug configuration and try again (staff diagnosis).
  • A thread whose context could not be read. EurekaLog 7.11.1.0, Delphi 11, Win32 on Windows 11 failed while dereferencing a thread-information-block address during stack collection. Support asked about the debugger and security software because either could affect thread state; the record does not establish antivirus as the cause. The expected consequence outside the debugger was missing background-thread stacks. Support offered 7.11.1.1, and the customer reported that this build seemed to fix the issue. That is a positive retest at a specific build boundary, not a general diagnosis for every unreadable thread context.
  • Execution that jumped somewhere invalid. A rare crash reported only by customers, with an address that matched itself in the message, gave a mostly blank stack, and the answer is candid: the tool is not magic, and when execution reaches an invalid location the walk can fail or produce nonsense; enable the assembler and CPU sections, without which the immediate return address and the registers are not in the file, and add a runtime log (staff diagnosis, staff proposal). The same answer reads the difference between two major versions as a hint that memory checks had made a pre-existing bug in the code visible.
  • Code that is not a module. Dynamic, generated code that is not registered in the platform's function tables cannot be unwound properly, and the tool's attempt to identify the module for such an address can raise and catch an access violation internally; support's answer is that this is expected, explains why the check is a guarded read rather than a validation call, gives the two real fixes (register the generated code, or stop processing handled exceptions in production), and adds a fact that changes how such reports are read: the handling path and call-stack-creation settings matter. In the recorded diagnosis, stack creation occurred at handling rather than at the original raise; do not generalise that to every configuration (chapter 11).

A retrieval freeze with a successful epilogue. A separate Win32 multithreading case initially reported an invalid-pointer failure under concurrent exception activity. The customer corrected the affected EurekaLog version to 7.12.0.462 beta 5; the initial form had named a newer version incorrectly. After an upgrade offer for 7.13.4.0, the customer explicitly confirmed that the invalid-pointer problem was resolved, but reported a separate freeze during unsynchronised stack retrieval. Internal logging changed the timing enough that it could hide the earlier reproduction.

The customer later corrected the failure location: the remaining freeze happened while fetching StackTrace or ExceptionManager.Info, not at the original raise. A 7.14 retest still froze. After support offered 7.14 Update 1, repeated customer runs under Delphi 2010 and Delphi 13 reported both no freezes and available call-stack information. That is a positive retest of the sample workload, not evidence that concurrent stack collection can never fail. Preserve the two problems and two retest stages instead of treating the entire exchange as an unresolved raise-time hang or blaming the customer's logger.

27.8 Entries that are not calls, and calls that look wrong

  • Special entries. A tool that parsed stacks for years started seeing entries that were not routine names; one documented cause is that the tool collapses a repeating area of the stack into a single marker instead of listing the repetition, and the marker can be recognised and filtered (staff diagnosis).
  • A wrong line rather than a wrong name. A viewer's jump to source once landed on the wrong line by a fixed offset (customer observation; no resolution is recorded). The documented conditions for the jump to work at all - matching sources, a running IDE that can find them, and the same user account and elevation level as the reporting application - are in chapter 20, section 20.4, and are the first things to check before assuming the symbol data is wrong.

27.9 What a solved stack needs

  1. The module was post-processed and the analysis of the image agrees; a stack of addresses with no post-processing is not a symbol problem.
  2. Debug information was produced and shipped: compiler debug information on, map generation on, no directive switching it off in a file, debug versions of the runtime's own units if you want their lines - and everything you want named, including the packages and third-party binaries whose frames matter, built with it.
  3. A reader exists for the format you actually ship, and it is included or deployed; for a separate symbol file, that the file travels with the application.
  4. The detail level shows what you have. Entries without debug information can be hidden by configuration; a one-line stack is worth checking against that option before anything else.
  5. The tracing method suits the artifact: frames where precision and speed matter, RAW where the call chain must survive a missing frame - and never RAW for leak reports in a release build.
  6. The platform's limits are known: which C++ Builder configuration can produce line numbers, and that a module you do not build cannot be made to answer. The report should also carry the sections that help when the stack does not - assembler, CPU registers and your own runtime log, the documented way to reconstruct what the stack could not say.

27.10 Where the evidence stops

There are real configuration successes here: a reporter found debug information disabled in the release target, and another obtained names after including the injected-format reader. Those do not prove that every one-line stack is a configuration mistake.

The 7.11.1.1 thread-context case has a positive customer retest. The 7.12.0.5 Hotfix 5 case also has a positive retest, but it followed an upgrade as well as configuration advice. The package-symbol checklist in 7.11.1.10 Beta 10 has no retest. The separate concurrent retrieval case has a positive 7.14 Update 1 retest under Delphi 2010 and 13. External symbol storage that failed to resolve and a viewer that jumped to the wrong source line remain unexplained in their respective exchanges.

When you compare a new report with an old one, record the exact EurekaLog build, IDE, bitness, tracing method, detail level and symbol files. Without that comparison, a restored stack demonstrates an improvement in that setup, not which individual change caused it.