1931557 - Half the total clobber build time is spent in geckoview:generateJNIWrappersForGeneratedDebug on a very powerful desktop

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Nazım Can Altınova [:canova]]

I have a powerful Ryzen Threadripper desktop that can build Firefox desktop on 4m 15s. When I try to build the Firefox for Android (geckoview), I get 9m 18s. It's a big difference and I could see that at least 3mins of it was spent during export. I'm pasting the resource usage profiles in case it might be useful:

Firefox Desktop (4m 15s): https://share.firefox.dev/4fN3XCx
Firefox Android (9m 16s): https://share.firefox.dev/3AxwwFf

Comment 1

[Markus Stange [:mstange]]

The export time is spent in gradle, it's doing "geckoview:generateJNIWrappersForGeneratedDebug".

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Markus Stange [:mstange] from comment #1)

The export time is spent in gradle, it's doing "geckoview:generateJNIWrappersForGeneratedDebug".

For me, this is incredibly fast, but I'll try it again. I'm very interested in why this is so slow on these beefy build boxes -- maybe I can add more profiling details to this step.

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

(In reply to Markus Stange [:mstange] from comment #1)

The export time is spent in gradle, it's doing "geckoview:generateJNIWrappersForGeneratedDebug".

For me, this is incredibly fast, but I'll try it again. I'm very interested in why this is so slow on these beefy build boxes -- maybe I can add more profiling details to this step.

Actually -- let me correct something from my memory. I think the slow part is compiling basically all of the GV JVM code in order to then process the .class files. I'm not sure how much information we can get from Gradle that would help here, but I'll see what can be done.

Comment 4

[Nick Alexander :nalexander [he/him]]

So, a little more thinking here:

1. I actually think what's happening is that there are two Android-y mach commands (assemble_app and generate_generated_jni_wrappers that are executing in serial (because they both want the same lockfile). If you look at the gradle-task span for the generated JNI wrappers, it's not long at all. So we're seeing serialization and some dodgy reporting more than slow task execution, per se.
2. I'm confused as to why everything is taking as long as it is. It's plausible that compiling all the JVM bits takes 2m43s -- there's a lot of code to process, and it's not all done in parallel -- but then why do we not see relevant gradle-task profile spans? That I would like to dig deeper into, but I have limited time.

:adhingra: could you dig further into these? The way that the generated JNI wrappers are produced is unusual: we invoke java "by hand". I'm not sure if that spawns a new JVM or does the work in the Gradle process itself; either way could stop us getting gradle-task profile spans, which is what we want to understand the times here. Thanks!

Comment 5

[Aaditya Dhingra [:adhingra]]

Thanks for the pointers! Looking through this now!

Profile: https://share.firefox.dev/3ZmUuMN

My profiler is a bit different though, my export step takes ~1.5 min. About 30 sec for each assemble-app, geckoview:generateJNIWrappersForGeneratedWithGeckoBinariesDebug and generate-sdk-bindings. This is gradle clean, clobber, bootstrap, and build on Non-artifact.

Another weird thing in mine different from yours is gradle:evaluate-project — fenix being run for each of the three commands in the end for ~8 sec each

Looking through it more, mainly on what the command is doing in the first 20 seconds

Comment 6

[Nick Alexander :nalexander [he/him]]

Great! I figured out a few things -- see my WIP patches in https://treeherder.mozilla.org/jobs?repo=try&revision=5228f85af4490eeb594e235907931fe28880e131, just the top three -- the ones labeled Bug GRADLE. With these, I get much faster Gradle times. The next step is to understand why configuring the root project is as slow as it is -- are we doing anything that should be lazy there? I would very much appreciate you testing these patches and expanding on them (or reviewing them and landing them more or less as-is), because I can't spend much time on this.

Comment 7

[Aaditya Dhingra [:adhingra]]

Attachment: Bug 1931557 - Don't invoke `mach environment ...` for `includeBuild` projects.

While investigating slow builds, I observed that we're evaluating
mozconfig.gradle -- and thus mach environment... for each
includeBuild project. It's not easy to share between (isolated)
projects in this manner -- see
https://github.com/gradle/gradle/issues/14697 -- but in this case, it
is possible.

N.b.: gradle.root evaluates to gradle when not there's no ambient
includeBuild -- see
https://github.com/gradle/gradle/blob/50b3c0bc19d8ef429ab559e9872cc6c351ad613b/subprojects/core/src/main/java/org/gradle/invocation/DefaultGradle.java#L162-L170.

For most of these uses, we're only configuring Maven repositories. In
the future, we can probably leverage more global configuration
approaches to avoid doing this "by hand" for each project, e.g.,
top-level pluginManagement and dependencyResolutionManagement
blocks. But for now, we'll just take the immediate win.

Comment 8

[Aaditya Dhingra [:adhingra]]

Attachment: Bug 1931557 - Configuration cache friendly listeners for build profile markers and add marker for time spent waiting for the gradle lockfile.

This commit moves toward configuration cache friendly listeners for
build profile markers. In addition, this approach gets events
for (isolated) projects that use includeBuild. Gradle is evolving
rapidly in this area; I expect there will be different, more
configuration cache friendly, ways to listen for task evaluation
events in the near future.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:rsainani, could you have a look please?

For more information, please visit BugBot documentation.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fa402aef91f9
Don't invoke `mach environment ...` for `includeBuild` projects. r=geckoview-reviewers,android-reviewers,tthibaud

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/fa402aef91f9

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e13f9040393
Configuration cache friendly listeners for build profile markers and add marker for time spent waiting for the gradle lockfile. r=florian,geckoview-reviewers,tthibaud

Comment 13

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/6e13f9040393

Comment 14

[Aaditya Dhingra [:adhingra]]

I think this can be closed now, please reopen if there is more work to be done here