Ensuring WASI-SDK follows standard expected behavior for debug information

[woodsmc]

Issue
WASI-SDK does not adhere to typical compiler convention on debug builds. It include debug information by default which presents a dramatic size in the resulting binary, inconsistent behavior, and confusion for developers new to the WASI ecosystem. Is it possible to address this, and the convention on using -g to control the presence of debug information in the final binary should be adhered to? -- I guess this might mean shipping both the debug and non-debug version of the WASI libraries?

Details
PR #422 turned on debug information in the WASI SDK. This means that applications which link against WASI get the DWARF information in their resulting compiled output. There is no means to avoid this. The debug information is always included, even if the developer doesn't want it. In the original PR, it was assumed that the developer would be familiar with the ecosystem and be able to remove this debug information with a post compilation step. However this isn't true for new entrants to the WASI ecosystem.

This behavior only happens when you link against the WASI SDK, if you remove the WASI SDK the DWARF information is not present, but can be controlled with the -g switch. Which is the behavior an experienced developer coming to WASI should expect. This behavior is inconsistent when compared to what happens when linking against WASI.

It is possible to remove the debug information at compilation time with the wl,--strip-debug linker command, but many new to the ecosystem are unaware that this additional step is now required to build a "standard" wasm module.

Impact on Developers new to WASI
While this change is great for debugging, but is unexpected behavior for developers who come to WASI from other platforms. The large size disparity (see graph below) is typically attributed to a WASI failing, not to the presence of debug information.

Building C Hello World for each

Developers new to the WASI world will not stick around long enough to try to understand why. They will simply reject WASI as generating bloated code, even though this is not actually the case. The net result is that new developers evaluating WASI for the first time often reject WASI outright, and instead seek alternatives.

Solution?
Is it possible to ship the WASI-SDK with debug & non-debug builds of the library, then link against the correct one if the -g debug flag is enabled?

Doing this would still provide the debug information we all need, but in a way which is consistent with expected behavior.

[sbc100]

-g is flag that normally only effect compilation, so using to to also select the libraries to link would actually be a departure from normal toolchain standards.

I'm pretty sure then when gcc and clang are linking they currently completely ignore -g.

I've said that emscripten, which I work on, does do crazy stuff like this and it will choose different libraries based on -O and -g flags. I'm not sure its good idea to copy that behavior in upstream clang though.

One alternative would be to provide two different sysroot, and/or two different SDKs, one for debug and one for non-debug. However, I also think its reasonable to just pass -Wl,--strip-debug or run the strip command after linking. Also not that a lot of users will or wasm will be using wasm-opt after linking, which will also strip debug info by default. I would even go so far as to say that if you are measuring code size at all you should be running wasm-opt first, since LLVMs default out contains over-wide LEBs (for relocations) which are pure overhead.

[alexcrichton]

Rust's solution to this problem is to pass the equivalent of -Wl,--strip-debug in release builds by default. This happens in Cargo, however, which has no equivalent in C/C++. Using rustc raw matches Clang, for example, and debuginfo is not stripped.

I'll also note that this issue is a duplicate of #460 which has its own discussion.

[woodsmc]

I guess the ambition here, is to give a developer an experience as close to what they would expect with other tooling. Currently they don't expect to see debug information included by default. As new arrivals in the wasm ecosystem they are not yet aware of the additional tooling / switches to remove the unexpected debug, they will instead walk away from wasm, or at the very least WASI. - Which is something I'm sure we'd all like to avoid.

It sounds like there are options which could provide a solution:

1. Automatically choosing libs based on -O and -g flags
   Pro: Gives the best user experience, allows for un-optimized libs and matching DWARF info, ensuring consistency of DWARF & compiled code.
   Con: Complex command line switch handling, which is only really needed for wasm targets. Pain to maintain. Also need to distribute multiple sysroots.

2. If -g is not specified or if -g0 is specified then add the -Wl,--strip-debug linker switch
   Pro: Similar to rust. No need to ship multiple binaries.
   Con: Some funky-ness in the switch handling which will probably need to be maintained / patched to upstream

3. Just ship two sysroots; If the developer wants debug, go grab the debug sysroot.
   Pro: Easy to implement. Maintains expectation for no-debug.
   Con: Developers debugging their application would need to know to switch sysroot.

[sunfishcode]

Another option is to use split dwarf. That is, to compile the sysroot with "g -gsplit-dwarf", which puts the bulk of the debug info in separate .dwo files, which don't get linked to executables. The downside is that the debugger needs to be able to find those files in order to use the debug info.

[sbc100]

What do other SDKs such as android NDK and iOS SDK do here? I imagine they simply do not ship debug info by default and getting debug info in system libraries would be some kind of manual extra step (either re-built or re-download a special sysroot?)

If my assumption is correct perhaps we don't try break new ground here and just remove the debug info by default too?

[yamt]

Another option is to use split dwarf. That is, to compile the sysroot with "g -gsplit-dwarf", which puts the bulk of the debug info in separate .dwo files, which don't get linked to executables. The downside is that the debugger needs to be able to find those files in order to use the debug info.

is there a standard convention for debuggers to find those files?

i vaguely remember debian debuginfo packages worked that way. but it was long ago i was working on them.

[whitequark]

@sbc100 I just checked the Android NDK and they ship .so files with debuginfo and .a files without. There are no .dwo files.

[sbc100]

Maybe we just drop the debug info then? How useful is the debug info to our users? Does anyone out there depend on it or find it useful (outside of wasi-libc developers themselves).

I personally don't so the original issue here as very impactfull since anyone doing size comparisons should be running wasm-opt and/or strip anyway. If they are not then the numbers they get will not be accurate in any case, so maybe we don't want to make it easier for folks to accidentally think they are looking at an optimized binaryen. Playing devils advocate: Maybe including debug info here by default could act as a clue/hint that the binaries should be optimized.

[whitequark]

Maybe we just drop the debug info then? How useful is the debug info to our users? Does anyone out there depend on it or find it useful (outside of wasi-libc developers themselves).

I last used it today while debugging a stack overflow. It was informative (pointed me where I wanted) albeit did not help me diagnose the root cause. I have a weak preference to keep it (it's not hard to rebuild a wasi-sdk sysroot, but it's pretty useful to have it by default and IMO the cost is negligible).

[SingleAccretion]

I would be in favor of keeping the debug info as well.

1. The consistency argument is a compelling one. However, with this change alone we won't achieve consistency since the name section will still be produced by default (you still need --strip-all to remove it).
2. While it may not be too hard to build a custom wasi-libc, building a custom libc just to get line numbers on a crash is not the native experience either (granted, the native experience, at least on some Linux distros, is not the best it could be either). It makes one scenario a bit better at the cost of another scenario being worse.

By the way, if we were considering a clang driver change here, we could make the argument that absence of -g should imply -Wl,--strip-all (or strip-debug + "strip name section"). That would make the feature simpler to explain from a user perspective: no -g => no debug info at all, -g => all debug info. However, that would be inconsistent with the native Unix experience as well.

Also, I see split DWARF (.dwo) was mentioned above. Split DWARF is not a DWARF distribution strategy. It exists to speed up large links (by moving the relocation work into the debugger, which I guess can do it in a more on-demand fashion).

[woodsmc]

I'm guessing that the debug build of WASI turned off all optimizations, as this leads to the best debug experience. Traditionally there is limited value in a highly optimized binary with debug information. I guess that's true here.??

I know you guys understand I lean toward disabling debug - let me explain why. By turning on the debug information by default the WASI SDK did three things:

1. It broke build tools and potentially CI/CD pipelines that relied on it - these now need to be updated with additional flags, assuming optimized -O3 building with -g debug information turned on. It is more likely that it necessary to rebuild the sysroot for production, - I'm guessing we have -O0 -g debug build. Move over, these are WASI SDK version dependent changes, catching people out when they update to newer versions of WASI.

2. It gives new developers / adopters of Wasm/WASI a larger file size, and sub optimal performance in general (assuming no optimization for debug) - a perception of cost which isn't reality.

3. Introduces a non-standard / expected behavior; developers don't expect debug information unless they request it. They don't expect it to be there

The implications
This results in existing users needing to reconfigure their CI/CD build chains, and new adopters being discouraged from taking the platform at all. The lack of stability adds cost to existing users, and the size concerns add cost for new users to investigate and resolve the issue - if they ever last that long. This will cause people to drop Wasm as a solution path prematurely, slowing adoption.

Debug
I think we folks in this repository and commenting on the issue are a rare-bread. We're the folks happy to poke around with WASI and are pretty familiar with it. We're often doing work with it / to it. I'd say we're not like most application developers, or early adopters of Wasm/WASI. It feels like designing for what we need, is not designing for what most developers need.

Most teams need the predictability of the tool chain. IMO: Keeping the debug is designing for the smaller set of developers (us). Turning it off provides stability in the tools, predictability in tooling behavior, and consistency in binary size output, for everyone else.

[whitequark]

I feel that you have not arrived at the root cause here. The problem is that you won't get the smallest binary size without --strip-all, so developer education is necessary regardless of whether debug information is included.

[woodsmc]

I feel that you have not arrived at the root cause here. The problem is that you won't get the smallest binary size without --strip-all, so developer education is necessary regardless of whether debug information is included.

That's playing a game of absolutism, which isn't the point I am raising. I'm not suggesting that developers need the absolute tiniest binary. I'm pointing out that the gigantic jump in binary size and change in tool chain behavior is the problem, a change which impacts everyone for the benefit of a minority.

I have seen first hand a development team new to Wasm reject WASI wholesale, and almost reject Wasm because of this single change. I am trying to mitigate this impact for the benefit of us all. - The team in question rejected education, claimed the tooling was immature, and it shouldn't be so hard to get something reasonable out of it, and refused to spend the cash (developer time), and simply moved on.

If we are designing tooling for the wider ecosystem then asking ourselves what we should do with the tooling is probably not the best way to go. I'll freely admit that many would say that basing this decision on the impact on a single team isn't good enough either. My concern is that this team's feedback is more of a barometer of impact, I'd certainly welcome more data here. From what I've seen the longer term impact on the overall ecosystem outweighs the shorter term benefits many on this forum (including myself) might derive. Personally, I am happy to take the impact of building a debug WASI if it helps the wider ecosystem thrive.

Based on my experience, once a developer is hooked into the tooling and the ecosystem, then developer education is possible - and I'm all for it... but they've got to see the value in investing first...

[yowl]

Slightly in favour of dropping it, size comes up a lot. Don't like that people (educated or not) are put off for something that would not be that contentious to drop.

[sbc100]

I have seen first hand a development team new to Wasm reject WASI wholesale, and almost reject Wasm because of this single change. I am trying to mitigate this impact for the benefit of us all. - The team in question rejected education, claimed the tooling was immature, and it shouldn't be so hard to get something reasonable out of it, and refused to spend the cash (developer time), and simply moved on.

If we are designing tooling for the wider ecosystem then asking ourselves what we should do with the tooling is probably not the best way to go. I'll freely admit that many would say that basing this decision on the impact on a single team isn't good enough either.

I have to say that does sounds a little extreme. I would not imagine that its common for folks to be that put off complete by the need to add a linker flag to strip debug into (or even the need to install a external tool, wasm-opt, to get the most optimized binaries). I would imagine there are much bigger roadblocks and unexpected behaviors when switching from the normal desktop toolchain to wasi-sdk. I don't mean to discount your anecdotal evidence, but I am very surprised by it.