New feature flag to allow for stack backtrace/unwind #8703

mcourteaux · 2025-08-02T17:32:01Z

I implemented this as a target feature flag. This way we can still disable this for actual performance optimized releases of software.

We could use Feature::Debug as another reason to enable this LLVM flag.

Fixes #8440
Fixes #8702
Fixes #8754

mcourteaux · 2025-08-02T18:02:28Z

@alexreinking Can you fix this? https://buildbot.halide-lang.org/master/#/builders/71/builds/223/steps/9/logs/stdio

alexreinking · 2025-08-02T18:09:01Z

@alexreinking Can you fix this? https://buildbot.halide-lang.org/master/#/builders/71/builds/223/steps/9/logs/stdio

I'm working on it -- sorry, I figured the weekend would be an OK time to mess with the bots.

mcourteaux · 2025-08-02T18:24:18Z

Aha hahahah! Carry on. 🫡 Lemme know once you think it's fixed, then I'll retrigger the bots. Thanks Alex!

slomp · 2025-08-02T20:46:32Z

src/CodeGen_Internal.cpp

    for (const char *s : {"halide_use_soft_float_abi",
-                          "halide_use_pic"}) {
+                          "halide_use_pic",
+                          "halide_keep_frame_pointer"}) {


I very much prefer the term "keep frame pointer", but in being consistent with compilers out there, should we change to "no_omit_frame_pointer"?

slomp · 2025-08-03T16:23:40Z

I just gave the fn.addFnAttr("frame-pointer", "all") a try on Windows, and it did not give me what I was expecting.
Still only get the FunctionName_par_for_output__1_s0_y_y, and no sign of any of its callers (halide_do_task, halide_do_par_for, the "FunctionName" Halide interface, and the application code that calls it).

In samples where there's no Halide code executing, but is in the Halide runtime, the stack unwind stops right there at the Halide runtime (the halide_mutex_lock you see there is not actually from the Halide runtime, but from the detour I installed):

This seems to be more indicative that we also need to emit frame pointers in the runtime as well.

shoaibkamil · 2025-08-04T18:04:48Z

I wonder if we need to add -fno-omit-frame-pointer here:

Halide/src/runtime/CMakeLists.txt

Line 160 in 4d8c660

-O3

slomp · 2025-08-04T18:43:25Z

Oh, and we are also purposedly stripping unwind information from the runtime with -fno-unwind-tables:

Halide/src/runtime/CMakeLists.txt

Line 165 in 4d8c660

-fno-unwind-tables

I think unwind tables are there to assist unwinding callstacks (maybe exceptions too?) when frame pointers are omitted.

abadams · 2025-08-04T18:59:42Z

Compiler flags in the cmakefile should be irrelevant here, if we do things right. There are other ways to build libHalide, so these should be controlled in code, not in the build. These flags should just set the initial state of the function attributes in the runtime functions, but they should get clobbered by CodeGen_LLVM.cpp line 629.

These functions are currently marked nounwind because it's impossible to throw exceptions safely through Halide-generated code (we'd leak all over the place), and according to the llvm docs, nounwind is for functions you can't throw exceptions through. There's another attribute "uwtable" which indicates "the ABI being targeted requires that an unwind table entry be produced for this function even if we can show that no exceptions passes by it." We could try adding that, with fn.addFnAttr("uwtable") in CodeGen_Internal.cpp in set_function_attributes_from_halide_target_options.

slomp · 2025-08-04T19:48:23Z

Yeah, I was doing some reading, and found out that the unwind tables for callstack unwinding just piggy-backs on the infrastructure arounf exception handling .eh_frame sections, which we can't rely on anyway.

@shoaibkamil I tried adding -fno-omit-frame-pointer to the runtime flags, but still no stack frames past the first runtime symbol that shows up.
Not sure what else we may need to change to ensure we emit frame pointers for things generated by Halide::compile_standalone_runtime().

mcourteaux · 2025-08-04T22:26:42Z

I just gave the fn.addFnAttr("frame-pointer", "all") a try on Windows, and it did not give me what I was expecting.

We could add those flags for the runtime functions too. I'll try that tomorrow and push it. You can try again. I'll ping you when I did this.

abadams · 2025-08-04T23:30:24Z

Working with Marcos, the thing that worked on windows was:

    fn.addFnAttr(llvm::Attribute::getWithUWTableKind(module.getContext(), llvm::UWTableKind::Default));

Turns out windows x64 doesn't really do frame pointers at all.

mcourteaux · 2025-08-05T07:24:46Z

Turns out windows x64 doesn't really do frame pointers at all.

What does that mean? The windows debugger ignores them for back tracing the call stack?

I see two options, and the "New Proposal" would get my preference.

Backwards-compatible way forward

What do we do with this flag? Unwind tables are big to store in the .eh_frame section. This should probably not be the default in Halide? Thoughts? Did you compare file size before and after?

I'm assuming Unwind Tables generated by LLVM would replace the need for the frame pointer. This is kinda pity, because on Linux, it does suffice to produce a backtrace.

So do we keep both flags as feature flags? KeepFramePointer/NoEmitFramePointer and UnwindTables?

New Proposal

I'm sort of thinking that the Debug feature flag should enable the required flags to make it debuggable. But right now, we use that for verbose logging from the runtime. I'm thinking that perhaps we enable all the verbose debug logging when enabling a new VerboseDebug/VerboseRuntime feature flag, and have the unwind-tables/frame-pointer stuff be influenced by Debug. That would be my preferred way forward. I never use the Debug flag right now, as it just spams so much stuff in the console, that it's not useful unless you're working on Halide GPU backends itself. With my proposed flags, it would start to make sense to enable the Debug feature flag in your development/debug builds of your software/application for increased debug-ability.

@slomp @shoaibkamil @abadams @alexreinking @steven-johnson @vksnk

slomp · 2025-08-05T16:51:10Z

What does that mean? The windows debugger ignores them for back tracing the call stack?

According to the MSVC compiler: "/Oy enables frame-pointer omission and /Oy- disables omission. In x64 compilers, /Oy and /Oy- are not available"
https://learn.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-omission?view=msvc-170

Going a bit deeper into the Windows 64 ABI, for proper unwinding, the PE image must include pdata and xdata sections that provide the necessary information. No frame pointers are needed.
https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170
(I think this is the same mechanism they use for SEH (structured exception handling), the Windows-equivalent to synchronous signals/traps).

Now, whether or not providing frame pointers on top of that will work and have benefits, is unclear. I am sure third-party libraries could use them to do the unwind, but I can't find much about the behavior of the "Windows unwinder" when frame pointers are used.

slomp · 2025-08-05T16:55:43Z

I'm sort of thinking that the Debug feature flag should enable the required flags to make it debuggable.

The thing is: callstacks are also invaluable for profiling, and profiling Debug modules is not ideal.

What do we do with this flag?

The safe thing to do for now is for unwinding to be "opt-in".

We would need to do some research to see if the impact of adding .eh_frame sections for Halide code and runtime modules is either onerous or negligible. The same goes for frame-pointer (since this works on Linux, and probably on Mac too).

abadams · 2025-08-05T17:49:47Z

My inclination would be to have a single flag that turns on frame pointers and unwind tables, and call it something to do with backtraces or call stacks. EnableBacktraces? PreserveCallStack?

…efault in that case. Also enable that feature when Target::Feature::Debug is set. Cleanup some helper functions.

mcourteaux · 2025-08-06T08:50:47Z

Update: changed the flag name to EnableBacktraces and added the UWTable fn attribute, and additionally also enable this when Debug feature flag is set. Instead of what @abadams wrote, I used this helper function:

https://llvm.org/doxygen/IR_2Function_8h_source.html#l00687

alexreinking · 2025-08-06T12:54:44Z

That correctness_register_shuffle failure seems specific to LLVM 21, which we weren't testing until this weekend.

slomp · 2025-08-06T13:51:15Z

src/CodeGen_Internal.cpp

    fn.addFnAttr("target-features", mattrs);
+    if (enable_bt) {
+        fn.addFnAttr("frame-pointer", "all");
+        fn.setUWTableKind(llvm::UWTableKind::Default);


Setting both frame pointer and unwind tables is fine for now, but I'd add a comment mentioning that having both set is not necessary, and which one is more appropriate depends on the system/arch/abi.

Well both is nice, because unwinding is very fast if you have frame pointers enabled. Unwinding via unwind information in .eh_frame is very slow in comparison. Although I don't think it would be a big deal, as the callstack depth in Halide routines is never very big. So perhaps its better to only rely on the unwind table...?

At this stage, we don't need to overthink too much, since it's opt-in.
Have both enabled, and we can tune things later, once we start using and having a better understanding of the implications in practice.

mcourteaux · 2025-08-06T19:30:37Z

@slomp is this working for you, on Windows?

slomp · 2025-08-06T21:48:11Z

@slomp is this working for you, on Windows?

Yes, just tested the fn.setUWTableKind(llvm::UWTableKind::Default); and it worked.

slomp · 2025-08-13T03:17:16Z

Where do we all stand with this one?
I'm inclined in merging, we just need to all agree on the name/label of the feature flag.
(I'm OK with the current "enable_backtraces".)

mcourteaux · 2025-08-13T07:25:27Z

I'm also in favor of merging as is. Waiting for approval.

mcourteaux · 2025-08-14T14:16:15Z

@abadams @alexreinking @slomp Approval to land? I'm juggling PRs.

abadams · 2025-08-14T22:37:18Z

src/runtime/HalideRuntime.h

-    halide_target_feature_no_bounds_query,  ///< Disable the bounds querying functionality.
+    halide_target_feature_jit = 0,            ///< Generate code that will run immediately inside the calling process.
+    halide_target_feature_debug,              ///< Turn on debug info and output for runtime code.
+    halide_target_feature_enable_backtraces,  ///< Keep the frame pointer in tact in functions produced by LLVM.


There should be something about unwind tables. Maybe "Preserve frame pointers and include unwind tables to support accurate backtraces for debugging and profiling."

abadams

looks good to me with a minor comment about the description of the feature flag

alexreinking · 2025-08-15T01:22:24Z

@mcourteaux - go ahead and make the documentation tweak @abadams suggested and then just merge.

alexreinking

Needs a one-line change for Python 😭

src/Target.h

alexreinking

Awesome, let's get this in

mcourteaux requested a review from abadams August 2, 2025 17:32

mcourteaux mentioned this pull request Aug 2, 2025

Debug symbols for LLVM-generated runtime objects #8632

Open

mcourteaux requested a review from halidebuildbots August 2, 2025 17:40

mcourteaux added enhancement New user-visible features or improvements to existing features. usability labels Aug 2, 2025

slomp reviewed Aug 2, 2025

View reviewed changes

slomp mentioned this pull request Aug 2, 2025

Allow users to preserve frame pointers in Halide-generated code #8702

Closed

New feature flag to keep the frame pointer for LLVM-generated functions.

1868183

mcourteaux force-pushed the keep-frame-pointers branch from 61ea8e1 to 4246ad3 Compare August 6, 2025 08:32

Rename KeepFramePointer to EnableBacktraces, and add UWTableKind to d…

f3dc266

…efault in that case. Also enable that feature when Target::Feature::Debug is set. Cleanup some helper functions.

mcourteaux force-pushed the keep-frame-pointers branch from 4246ad3 to f3dc266 Compare August 6, 2025 08:48

Clang-tidy.

13d171d

mcourteaux requested a review from slomp August 6, 2025 09:53

slomp reviewed Aug 6, 2025

View reviewed changes

slomp changed the title ~~New feature flag to keep the frame pointer for LLVM-generated functions.~~ New feature flag to allow for stack backtrace/unwind Aug 6, 2025

abadams reviewed Aug 14, 2025

View reviewed changes

abadams approved these changes Aug 14, 2025

View reviewed changes

alexreinking requested changes Aug 15, 2025

View reviewed changes

src/Target.h Show resolved Hide resolved

alexreinking self-requested a review August 15, 2025 01:38

alexreinking approved these changes Aug 15, 2025

View reviewed changes

alexreinking mentioned this pull request Aug 15, 2025

The Python bindings are out of sync with Target::Features #8754

Closed

Update Python bindings Target flags and generator emit options.

59b7375

slomp approved these changes Aug 15, 2025

View reviewed changes

alexreinking merged commit 6709900 into halide:main Aug 15, 2025
5 of 17 checks passed

BrewTestBot mentioned this pull request Sep 16, 2025

halide 21.0.0 Homebrew/homebrew-core#244220

Merged

New feature flag to allow for stack backtrace/unwind #8703

New feature flag to allow for stack backtrace/unwind #8703

Uh oh!

Conversation

mcourteaux commented Aug 2, 2025 • edited by alexreinking Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcourteaux commented Aug 2, 2025

Uh oh!

alexreinking commented Aug 2, 2025

Uh oh!

mcourteaux commented Aug 2, 2025

Uh oh!

slomp Aug 2, 2025

Choose a reason for hiding this comment

Uh oh!

slomp commented Aug 3, 2025

Uh oh!

shoaibkamil commented Aug 4, 2025

Uh oh!

slomp commented Aug 4, 2025

Uh oh!

abadams commented Aug 4, 2025

Uh oh!

slomp commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcourteaux commented Aug 4, 2025

Uh oh!

abadams commented Aug 4, 2025

Uh oh!

mcourteaux commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backwards-compatible way forward

New Proposal

Uh oh!

slomp commented Aug 5, 2025

Uh oh!

slomp commented Aug 5, 2025

Uh oh!

abadams commented Aug 5, 2025

Uh oh!

mcourteaux commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexreinking commented Aug 6, 2025

Uh oh!

slomp Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

mcourteaux Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

slomp Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

mcourteaux commented Aug 6, 2025

Uh oh!

slomp commented Aug 6, 2025

Uh oh!

slomp commented Aug 13, 2025

Uh oh!

mcourteaux commented Aug 13, 2025

Uh oh!

mcourteaux commented Aug 14, 2025

Uh oh!

abadams Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

abadams left a comment

Choose a reason for hiding this comment

Uh oh!

alexreinking commented Aug 15, 2025

Uh oh!

alexreinking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexreinking left a comment

Choose a reason for hiding this comment

mcourteaux commented Aug 2, 2025 •

edited by alexreinking

Loading

slomp commented Aug 4, 2025 •

edited

Loading

mcourteaux commented Aug 5, 2025 •

edited

Loading

mcourteaux commented Aug 6, 2025 •

edited

Loading