1442253 - Binary from optimised Linux build is 44M

Status: RESOLVED FIXED

(Testing :: geckodriver, defect, P3)

Description

[Andreas Tolfsen ❲:ato❳]

nalexander noted in https://bugzil.la/1441656#c3 that the
geckodriver binary produced by an optimised Firefox build in central
is 44M.  I confirmed that this is also the case on Linux:

> % du -shL obj-x86_64-pc-linux-gnu/dist/bin/
> 44M	obj-x86_64-pc-linux-gnu/dist/bin/geckodriver

If we compare this to a "cargo build --release" build, we can see
that the expected size is somewhere close to ~8.7M:

> % du -sh testing/geckodriver/target/release/geckodriver
> 8.7M	testing/geckodriver/target/release/geckodriver

This bug is not related to https://bugzil.la/1392313, which aim is
to _further_ reduce the binary size to a more managable level beyond
8.7M.

Comment 1

[Andreas Tolfsen ❲:ato❳]

This bug is obviously preventing us from distributing release
binaries produced by TaskCluster.  Do you have any ideas or pointers
where we should look as to why the build produced by "./mach build
testing/geckodriver" (ensure you have ac_add_options --enable-geckodriver
in your mozconfig first) is so vastly larger in size than those of
"cd testing/geckodriver; cargo build"?

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to Andreas Tolfsen 「:ato」 from comment #1)
> This bug is obviously preventing us from distributing release
> binaries produced by TaskCluster.  Do you have any ideas or pointers
> where we should look as to why the build produced by "./mach build
> testing/geckodriver" (ensure you have ac_add_options --enable-geckodriver
> in your mozconfig first) is so vastly larger in size than those of
> "cd testing/geckodriver; cargo build"?

Does your mozconfig have --enable-debug-symbols or the like turned on?  That would probably account for a large chunk of it, since I believe `cargo build --release` wouldn't include debug symbols by default.

Regardless of whether that helps, looking at `readelf -SW` output for each binary would be interesting.

Comment 3

[Nathan Froyd [:froydnj]]

Also, the default rustc opt level for `./mach build` is probably somewhat different than `cargo build --release`, and the latter probably includes LTO?  Sticking `ac_add_options --enable-release` in your mozconfig might help`.

Comment 4

[Nathan Froyd [:froydnj]]

Oh, and if you're doing this locally, incremental compilation is probably turned on (?), so you would want to ensure that's turned off as well, as incremental compilation trades size of binaries for better compilation times.  (Incremental compilation should be turned off on automation builds.)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I noticed the same yesterday while working on bug 1493948 and comparing the binary sizes between different platforms. As it looks like this issue only exists on Linux 32/64 but not on MacOS nor Windows:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=2929c7f9a5e06800dbc59a8bf6bd2f1790e103fa

Just click on any of the Fetch-geckodriver jobs, and select the geckodriver artifact.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: readelf output for "cargo build --release"

$ ls -l target/release/geckodriver
-rwxrwxr-x 2 henrik henrik 6141392 Feb 22 11:12 target/release/geckodriver

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: readelf output for "./mach build testing/geckodriver"

$ ls -l obj-x86_64-pc-linux-gnu/testing/geckodriver/x86_64-unknown-linux-gnu/release/geckodriver
-rwxrwxr-x 2 henrik henrik 43927704 Feb 22 16:17 obj-x86_64-pc-linux-gnu/testing/geckodriver/x86_64-unknown-linux-gnu/release/geckodriver

Comment 8

[Nathan Froyd [:froydnj]]

It looks like that binary has all the debug info in it, which is ~1/3-1/2 of the binary.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note that using ac_add_options --enable-release doesn't make a difference. Also when I compile with cargo incremental compilation is turned off.

(In reply to Nathan Froyd [:froydnj] from comment #8)

It looks like that binary has all the debug info in it, which is ~1/3-1/2 of the binary.

I wonder why that is not stripped off. But even then the binary size would be still 5x that large as with cargo build --release.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Also why does that only happen for Linux build jobs but not Mac and Windows? Maybe we missed to add something to the build configuration for this platform?

Comment 11

[Nathan Froyd [:froydnj]]

(In reply to Nathan Froyd [:froydnj] from comment #8)

It looks like that binary has all the debug info in it, which is ~1/3-1/2 of the binary.

This comment applied to the readelf output for the cargo --release binary. You can see the debug info starting:

  [28] .bss              NOBITS          000000000055fd20 35fd20 0002a8 00  WA  0   0  8
  [29] .comment          PROGBITS        0000000000000000 35fd20 000057 01  MS  0   0  1
  [30] .debug_pubnames   PROGBITS        0000000000000000 35fd77 027658 00      0   0  1

...and continuing on from there. That 0x35fd77 is the start of .debug_pubnames; basically everything prior to that is what your binary size is. So about 3.5MB (plus a little extra for the .{sym,str}tab sections later on).

Whereas the mozilla-central geckodriver has:

  [30] .bss              NOBITS          000000000056fd10 36fd10 0002a8 00  WA  0   0  8
  [31] .comment          PROGBITS        0000000000000000 36fd10 0000a9 01  MS  0   0  1
  [32] .debug_pubnames   PROGBITS        0000000000000000 36fdb9 21ec85 00      0   0  1

Notice that .debug_pubnames starts at 0x36fdb9, so about the same place as the cargo --release version. But the debug info sections are much bigger. Compare the cargo --release bits:

  ...
  [39] .debug_ranges     PROGBITS        0000000000000000 51ce01 0497a0 00      0   0  1
  [40] .symtab           SYMTAB          0000000000000000 5665a8 01bd08 18     41 2728  8

where .symtab starts at 0x5665a8, to the mozilla-central version:

  ...
  [41] .debug_ranges     PROGBITS        0000000000000000 2579e20 3f5f60 00      0   0  1
  [42] .symtab           SYMTAB          0000000000000000 296fd80 01bc18 18     43 2741  8

where .symtab starts at 0x296fd80, which is a significant difference in the amount of debug info. We compile with more debug information in m-c versus cargo --release, so that's probably where the size difference is coming from on Linux.

On Mac, at least, debug information can be stored in separate files, so if you hunted around for those, you'd probably notice similar size differences. Same thing on Windows.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I see. Do you know of a way to instruct the build to not include the debug info but store it in a separate file on Linux? Or are we forced to strip the binaries afterward?

Maybe we should simply disable debug info at all for the geckodriver opt builds in mc, only leave them enabled for debug builds?

Comment 13

[Nathan Froyd [:froydnj]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #12)

I see. Do you know of a way to instruct the build to not include the debug info but store it in a separate file on Linux? Or are we forced to strip the binaries afterward?

There's a GCC option (clang too?) called...-gsplit-dwarf, I think? I have no idea whether there's anything comparable for Rust.

I know we strip target binaries (see ENABLE_STRIP uses in config/rules.mk), but it looks like we don't do anything similar for host binaries.

Maybe we should simply disable debug info at all for the geckodriver opt builds in mc, only leave them enabled for debug builds?

We could do that via separate Rust flags for target and host builds (and maybe even programs vs. libraries, depending).

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so I think it will be better to wait for bug 1534533 then which will move building geckodriver to its own toolchain build task.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I triggered a new geckodriver toolchain job, which we will make use of for releasing geckodriver in the future:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=7004b8779a36084c898634e5e31d8f406815d68a&selectedJob=264006705

And the geckodriver binary for Linux is 6.6 MB only now!