1744336 - substitute-local-geckoview is very slow (5min+) when run in ac due to mach calls

Status: RESOLVED FIXED

(GeckoView :: General, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

On my machine, builds with ac + geckoview dependency substitution take a long time. For an incremental ./gradlew :browser-state:assembleDebug, it's 5 minutes compared to less than 10 seconds without substitution.

Currently, substitute-local-geckoview.gradle is being applied for every subproject – in ac, there ~113 subprojects. In my proof-of-concept, I found the root cause is that calling out to ./mach is slow (because we have to start the python interpreter each time) – see https://github.com/mozilla-mobile/android-components/issues/11359#issuecomment-985802659.

I think the solution is to execute the mach command to load the mozconfig once and pass that object into a function that is run on each subproject to do the dependency substitution.

See https://github.com/mozilla-mobile/android-components/issues/11359 for the downstream issue in ac.

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1744336 - add loadMozconfig + indent existing code. r=agi

This change breaks the existing code. It's intended as a pure indentation change
that is easy to review and makes the subsequent commit, where we make functional
changes, easier to read.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1744336 - load mozconfig only once in substitute-local-geckoview. r=agi

Root cause analysis: in the changed implementation, the
substitute-local-geckoview script is called for every subproject in
android-components. Since this script calls a mach command, we're forced to wait
on the python interpretter for each a-c subproject (~113), which is very slow.

This patch addresses the problem by caching the output of loadMozconfig, which
calls the mach command, on the first run and using the cache on subsequent
invocations of the script.

Locally, with non-robust testing, I saw incremental build times decrease from
over five minutes to between 5-30 seconds.

This solution is ideal because it does not break downstream consumers. However,
it may not be the most correct solution because it's difficult to avoid writing
code that's not redundant if called multiple times. A more correct solution
would request that consumers call the script once and return a function for them
to call for each subproject.

Depends on D132849

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d7252afd27ee
load mozconfig only once in substitute-local-geckoview. r=agi

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d7252afd27ee