1307435 - Easy symbol generation optimization improvements

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

First part of work for bug 1307301.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1307435 - Generate symbols zip file concurrently;

Each archive takes 10+ seconds to generate. Split the archiving
part into separate make targets so this completes faster.

I'm not super thrilled about adding yet more make targets to the
root Makefile.in. We can refactor this later. For now, let's
take the quick win.

Review commit: https://reviewboard.mozilla.org/r/83274/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/83274/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1307435 - Change compression level of symbols files;

Currently, we use the gzip default of 6. Our history with zlib tells
us that reducing the compression level to 5 or 4 often yields
significantly faster operations while only sacrificing a little
storage. Measurement here shows similar results.

On libxul.so.dbg:

level      time      compressed
6          21.0s     231,045,158
5          15.8s     232,926,435
4          12.2s     237,587,011
3          11.1s     245,104,157

Changing the level from 6 to 4 increases the size of the compressed
file by 6,541,853 bytes, or 2.83%. But it saves ~10s from the long
pole of builds in automation. And that's just from libxul.

When you factor in all compressed files, this change has a significant
impact on symbol generation.

Before: 221s wall; 150s CPU; 311,424,856 bytes
After:  192s wall; 130s CPU; 318,085,885 bytes

That's on my machine, which has a 4.0 GHz CPU. CPU time savings in
automation will likely be more significant.

Review commit: https://reviewboard.mozilla.org/r/83276/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/83276/

Comment 3

[Gregory Szorc [:gps]]

ted: feel free to autoland if you r+, as I'll be on a plane later

Comment 4

[(not currently active) Ted Mielczarek]

Comment on attachment 8797600 [details]
Bug 1307435 - Generate symbols zip file concurrently;

https://reviewboard.mozilla.org/r/83274/#review85146

This seems like a straightforward win for the short-term. Longer-term maybe we should move the zip file generation into the Python script?

::: Makefile.in:331
(Diff revision 1)
>  	$(NSINSTALL) -D $(DIST)/$(PKG_PATH)
> +endif # MOZ_CRASHREPORTER
> +
> +.PHONY: symbolsfullarchive
> +symbolsfullarchive: generatesymbols
> +ifdef MOZ_CRASHREPORTER

Maybe you should just wrap these whole targets in the ifdef, and then have like:
```
ifdef MOZ_CRASHREPORTER:
buildsymbols: symbolsfullarchive symbolsarchive
else
buildsymbols:
endif
```

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8797601 [details]
Bug 1307435 - Change compression level of symbols files;

https://reviewboard.mozilla.org/r/83276/#review85148

Might be useful to look at tweaking the tar.bz2 generation in the OS X case. I know you've looked at the Windows PDB compression bit in the past.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8797600 [details]
Bug 1307435 - Generate symbols zip file concurrently;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/83274/diff/1-2/

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8797601 [details]
Bug 1307435 - Change compression level of symbols files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/83276/diff/1-2/

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5a4b248fce1f
Generate symbols zip file concurrently; r=ted
https://hg.mozilla.org/integration/autoland/rev/7cd613179479
Change compression level of symbols files; r=ted

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5a4b248fce1f
https://hg.mozilla.org/mozilla-central/rev/7cd613179479