Investigate shrinking code size by removing strings from builds

[fitzgen]

We suspect error strings are a major code size offender, and revamping wasmtime::Error to optionally (based on compile-time features) contain just error codes, instead of full strings, could improve code size.

And then for regular Wasmtime CLI builds, we would probably want something like

$ wasmtime --explain 1234
Error 1234: <one-line explanation>

<paragraph with more detail, if applicable>

Apparently the defmt and/or pw_tokenizer crates could be helpful here, although I have no experience with either of them.

[bjorn3]

For me CARGO_PROFILE_RELEASE_OPT_LEVEL=s CARGO_PROFILE_RELEASE_LTO=fat CARGO_PROFILE_RELEASE_STRIP=symbols CARGO_PROFILE_RELEASE_PANIC=abort cargo build -p wasmtime-c-api --release --no-default-features --features disable-logging produces a 955kb dylib on x86_64-unknown-linux-gnu. Of this the total rodata size is just a little over 60kb. This is the upper bound on the size of the error messages.

With panic_immediate_abort (CARGO_PROFILE_RELEASE_OPT_LEVEL=s CARGO_PROFILE_RELEASE_LTO=fat CARGO_PROFILE_RELEASE_STRIP=symbols CARGO_PROFILE_RELEASE_PANIC=abort RUSTFLAGS="-Zlocation-detail=none" cargo +nightly build -p wasmtime-c-api --release --no-default-features --features disable-logging -Z build-std=std,panic_abort --target x86_64-unknown-linux-gnu -Z build-std-features=panic_immediate_abort) the dylib is 531kb with only 34kb in rodata.

A quick look through the contents of the .rodata section shows that with panic_immediate_abort maybe half are error messages. A significant part of this however are not coming from wasmtime::Error but rather from various eprintln!() by wasmtime-c-api for unimplemented things, the crash message of the signal handler, various aborts in the internals of wasmtime, error messages produced by postcard, context messages for anyhow and things like that. As for the other half that are not error messages, they are things like cranelift flag names, names of libcalls and a whole lot of non-string data.

[bjorn3]

By the way there is currently more space taken up by relocations (.rela.dyn) than .rodata and the actual code takes up 77% of the binary, there are probably bigger things to worry about than error messages.

[hanna-kruppe]

The actual strings don't seem to be significant from those numbers. What might be more relevant use is heavy use of std::fmt and anyhow::Error, both of which create a whole lot of code in .text and (for PIC) relocations. It's difficult to estimate how much of a difference it makes without doing the hard work of switching over, but I've seen its traces often enough while digging through compiler output. Some specific points:

• With respect to std::fmt, even a trivial function that just returns format!("x = {x}") takes 102 bytes of code. It's not clear to me if the GOTPCREL relocations make it into the final .so but at least the &str pointers in the static data will do that. Both code size and relocations increase proportionally with the number of interpolated values and string pieces.
• anyhow builds heavily on string formatting, both directly with the bail!, ensure!, etc. macros and by converting every concrete error type into trait objects whose vtable includes Debug and Display impls, which are often full of more std::fmt code. (The latter is not really anyhow-specific, plain Box<dyn std::error::Error> also does this.)
• In addition, last time I checked anyhow unconditionally pulled in code for capturing and printing backtraces, which may negate some of the benefit of panic=abort and panic_immediate_abort.

As higher-level anecdata: I have a (non-public) project where I recently replaced all uses of anyhow::Error with a mostly API-compatible error type that only stores a String and a cause (recursively) internally, converts other error types to their Display outcome (never Debug), and doesn't support downcasting or backtraces. Despite still using std::fmt to some extent, this change alone saved about 80 KiB from an opt-level=s, lto=true, stripped binary on x86_64-unknown-linux-gnu.

[kornelski]

-Zfmt-debug=none may help remove code and strings used by fmt::Debug.