1747897 - 12.91% build times (Android) regression on Sat December 18 2021

Status: NEW

(Core :: General, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a build_metrics performance regression from push bc4c2236d4ddffab0f62bd27f8c5ea82ede756cd. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
13%	build times	android-5-0-x86_64	gcp taskcluster-projects/970387039909/machineTypes/custom-32-73728	1,378.74 -> 1,556.73

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Christian Sadilek [:csadilek]]

This seems to be caused by an increase in compilation times after the latest dav1d update, which added more files to the x86 source set. Since this only affects Android5/x86 build times I don't think this warrants backing out the patch.

I am not sure this is actionable either. Jon, can you confirm, double-check the build changes?

Comment 2

[Jon Bauman [:jbauman:]]

Attachment: dav1d-update-diffstat.txt

This looks right to me. I'm guessing the increase in build times is largely due to two newly added, fairly large assembly files:
- [third_party/dav1d/src/x86/itx_avx512.asm](https://hg.mozilla.org/integration/autoland/file/bc4c2236d4ddffab0f62bd27f8c5ea82ede756cd/third_party/dav1d/src/x86/itx_avx512.asm): 289 KBytes
- [third_party/dav1d/src/x86/mc16_avx512.asm](https://hg.mozilla.org/integration/autoland/file/bc4c2236d4ddffab0f62bd27f8c5ea82ede756cd/third_party/dav1d/src/x86/mc16_avx512.asm): 164 KBytes

As expected, these are both  x86-specific. No similarly large changes occurred in other architectures. Here's the breakdown of the magnitude of the changes in that last dav1d update:
```

Comment 3

[Jon Bauman [:jbauman:]]

Attachment: dav1d-update-diffstat.txt

This looks right to me. I'm guessing the increase in build times is largely due to two newly added, fairly large assembly files:

• third_party/dav1d/src/x86/itx_avx512.asm: 289 KBytes
• third_party/dav1d/src/x86/mc16_avx512.asm: 164 KBytes

As expected, these are both x86-specific. No similarly large changes occurred in other architectures. I've attached the breakdown of the magnitude of the changes in that last dav1d update

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I'm not informed about native development but regressing the build by 13% – ~2min (22.98 min to 24.95 min) – seems extreme for a dependency upgrade, especially when one of dav1d's aims is to be small. However, iiuc there are some discrepancies:

• this regression only affects the GCP builds (which I think we're migrating to? bug 1547111) and not the AWS build
• the regression only affects x86_64 and not other architectures

It could be something weird with the GCP builder and not something we should worry about affecting local devs, i.e. the regression may be less impactful or is a configuration issue and may go away on its own.

:jbauman, what do you think? Is there anything reasonable and worth our time we can do to address this?

(In reply to Christian Sadilek [:csadilek] from comment #1)

This seems to be caused by an increase in compilation times after the latest dav1d update, which added more files to the x86 source set. Since this only affects Android5/x86 build times I don't think this warrants backing out the patch.

To elaborate on who this impacts, iiuc the Android 5 builds are our primary Gecko/GeckoView builds so this build time regression may impact all x86_64 builds. At least amongst mobile devs, these builds are primarily used 1) in CI and 2) by developers testing on Android emulators. As such this probably isn't too impactful (though I wonder if incremental build times are affected).

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

To be more explicit, this seems low impact so I don't think it's worth spending a lot of time on or backing this out: I'm just curious what's possible.

Comment 6

[Jon Bauman [:jbauman:]]

:jbauman, what do you think? Is there anything reasonable and worth our time we can do to address this?

No idea, really. I've been shepherding the dav1d updates, but I'm not an expert in that codebase nor our build system. It does seem like a surprising increase, but I don't see that we're doing anything we obviously shouldn't.

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I'm assuming we're not going to address this and I assume it's not a great practice to leave this bug open: :jbauman, do you think we should WONTFIX this?

Comment 8

[Jon Bauman [:jbauman:]]

Seeing as we'll be landing an update to dav1d in bug 1754070 (hopefully imminently), it might be worth keeping this open until then and seeing if there's any change. That said, if it's basically the same, I'm not sure it's worth the effort to keep this issue.