1906365 - Don't use `dotgradle-online` environment for Firefox for Android Gradle tasks

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Nick Alexander :nalexander [he/him]]

While reviewing Bug 1893181, I discovered that all of the Android Gradle tasks are using an online environment (i.e., dotgradle-online): see https://searchfox.org/mozilla-central/rev/e4a222169f9f9b1d8f2f49e4066585b6b9c4dc58/taskcluster/android_taskgraph/job.py#85. Indeed, I see this called out in https://bugzilla.mozilla.org/show_bug.cgi?id=1824856#c79.

This is not what we want; we expect all build tasks to be offline and use pre-fetched dependencies. This ticket tracks making that happen.

Comment 1

[Nick Alexander :nalexander [he/him]]

jcristau: please dupe this if it's already tracked.

Jan-Erik: how do we make this better? My very preliminary thoughts, having not thought about this problem before: The Firefox build system already does non-trivial Python virtualenv management. As a consumer, it would make the most sense to be able to tell the Glean Gradle plugin "don't worry about environment management, I'll handle it". If the plugin accepted a virtualenv directory around https://github.com/mozilla/glean/blob/23b8d9e195a09d0a71aac6a4d08b494f59b305ef/gradle-plugin/src/main/groovy/mozilla/telemetry/glean-gradle-plugin/GleanGradlePlugin.groovy#L394 directly, then I expect tasks would be able to Just Work using the Firefox-managed python/pip environment. That would probably allow the in-tree/vendored glean_parser to be used directly rather than vendoring wheels, which I think is also part of the desired end state. But -- surely others have thought more deeply about this problem than have I; suggestions appreciated.

Nicolas: let's track work on this specific problem here, rather than in the Searchfox-specific tickets, since those aren't going to be particularly visible. Could you investigate whether what I say above is plausible? It occurs that you could symlink the directory above to one of the mach-managed Python environments and see what happens. And I see now you propose the same, essentially, in https://phabricator.services.mozilla.com/D215795?id=886406#7412945. Huzzah!

Comment 2

[Nicolas Guichard]

It occurs that you could symlink the directory above to one of the mach-managed Python environments and see what happens.

That works, here is a Try build with that: https://treeherder.mozilla.org/jobs?repo=try&revision=c774f53e62bcd6d00ecd948f6c51439a4f0a5e0f

Now I think a proper fix should be on the Glean Gradle plugin side, something like this
Then in the root build.gradle we would add allprojects.ext.gleanPythonEnvDir = <provided by mach>.
That would apply to both offline and online builds, and would enable us to remove the workaround for the concurrent Miniconda installs issue in android-gradle-dependencies.sh:27-30.
This is still 100% untested.

Comment 3

[Julien Cristau [:jcristau]]

There is some previous discussion in bug 1876587.

Comment 4

[Jan-Erik Rediger [:janerik]]

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

jcristau: please dupe this if it's already tracked.

Jan-Erik: how do we make this better? My very preliminary thoughts, having not thought about this problem before: The Firefox build system already does non-trivial Python virtualenv management. As a consumer, it would make the most sense to be able to tell the Glean Gradle plugin "don't worry about environment management, I'll handle it". If the plugin accepted a virtualenv directory around https://github.com/mozilla/glean/blob/23b8d9e195a09d0a71aac6a4d08b494f59b305ef/gradle-plugin/src/main/groovy/mozilla/telemetry/glean-gradle-plugin/GleanGradlePlugin.groovy#L394 directly, then I expect tasks would be able to Just Work using the Firefox-managed python/pip environment. That would probably allow the in-tree/vendored glean_parser to be used directly rather than vendoring wheels, which I think is also part of the desired end state. But -- surely others have thought more deeply about this problem than have I; suggestions appreciated.

Yeah, some form of specifying the environment from the outside would be good. Afair there were some other subtle issues with the plugin, especially given newer Gradle versions, so a bit of an overhaul might be in order when we do this anyway.

Comment 5

[BMO Github Automation]

Attachment: [mozilla/glean] Bug 1906365 - Gradle plugin: support using an external venv. (#2889)

Comment 6

[Nicolas Guichard]

Attachment: Glean Gradle plugin Pull Request

Comment 7

[Nicolas Guichard]

This is still 100% untested.

I have now tested the approach above in this Try run and it works. We can see that the gradle-dependencies task (using gradle --online) doesn't use Miniconda anymore and the Android Searchfox task (using gradle --offline) succeeds without supplying the glean_parser wheels.

Comment 8

[Nicolas Guichard]

Attachment: Bug 1906365 - Configure the Glean Gradle Plugin to use our vendored glean_parser. r=nalexander!

The Glean Gradle plugin now exposes ext.gleanPythonEnvDir for users to
point to an existing Python virtualenv where glean_parser is available.

Comment 9

[Nick Alexander :nalexander [he/him]]

I see that the Glean Gradle plugin changes are merged. As soon as there's a plugin release, could we get an updated patch that also changes https://searchfox.org/mozilla-central/rev/e4a222169f9f9b1d8f2f49e4066585b6b9c4dc58/taskcluster/android_taskgraph/job.py#85 and a try build? Let's get :jcristau's stamp as well and then we're good to go here -- at least, as far as I'm concerned.

Great job, Nicolas!

Comment 10

[Jan-Erik Rediger [:janerik]]

\o/
I'll let you know once we have that Glean release in. Might be this week as we have other pending stuff anyway.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Glean v60.4.0 was released this week. Is this good to proceed now?

Comment 12

[Nicolas Guichard]

Yes it should be, thanks for notifying me.
I just launched a Try push with the default dotgradle changed to offline as requested by Nick, I'll update the patch once it passed (or I've fixed the errors).

Comment 13

[Jan-Erik Rediger [:janerik]]

Bringing this up again. Can we land this now?

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ecbe936878e
Configure the Glean Gradle Plugin to use our vendored glean_parser. r=nalexander,geckoview-reviewers,android-reviewers,owlish

Comment 15

[Iulian Moraru]

Backed out 2 changesets (bug 1906365, bug 1893181) for causing mass android-all build bustages.

• Push with failures

• Failure log

• Backout link

[task 2024-08-15T16:15:31.933Z] To suppress this warning, add/update
[task 2024-08-15T16:15:31.933Z]     android.suppressUnsupportedCompileSdk=35
[task 2024-08-15T16:15:31.933Z] to this project's gradle.properties.
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] FAILURE: Build failed with an exception.
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] * What went wrong:
[task 2024-08-15T16:15:32.233Z] Could not determine the dependencies of task ':support-test:extractDebugAnnotations'.
[task 2024-08-15T16:15:32.233Z] > Failed to find Build Tools revision 34.0.0
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] * Try:
[task 2024-08-15T16:15:32.233Z] > Run with --stacktrace option to get the stack trace.
[task 2024-08-15T16:15:32.233Z] > Run with --info or --debug option to get more log output.
[task 2024-08-15T16:15:32.234Z] > Run with --scan to get full insights.
[task 2024-08-15T16:15:32.234Z] > Get more help at https://help.gradle.org.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] BUILD FAILED in 1m 27s
[task 2024-08-15T16:15:32.234Z] 10 actionable tasks: 10 executed
[taskcluster 2024-08-15 16:15:33.105Z] === Task Finished ===

Comment 16

[Nicolas Guichard]

fenix-debug was green in my Try run: https://treeherder.mozilla.org/jobs?repo=try&revision=b29dbe0885d774eb3e1ba55bccdabc00eeea073f&selectedTaskRun=QByr1m0oSs2a4jqKSKyR5g.0
But is red in with the landed tree: https://treeherder.mozilla.org/jobs?repo=autoland&revision=5c1c6d2d6993af568c148dba43cf5a89128d2dcf&selectedTaskRun=RjEbwSHJTSyrU94fMh9Xag.0

I think this interacted badly with something landed in-between, probably Bug 1880792 which updated the SDK version to 35. But the build still tries to use v34 in places because that's what AGP 8.4 defaults to:

[task 2024-08-15T16:19:56.060Z] FAILURE: Build failed with an exception.
[task 2024-08-15T16:19:56.060Z] 
[task 2024-08-15T16:19:56.060Z] * What went wrong:
[task 2024-08-15T16:19:56.060Z] Could not determine the dependencies of task ':app:compileFenixDebugJavaWithJavac'.
[task 2024-08-15T16:19:56.060Z] > Could not determine the dependencies of null.
[task 2024-08-15T16:19:56.060Z]    > Failed to find Build Tools revision 34.0.0

This is basically the same as Bug 1881001#c5 which prompted me to handle Bug 1881001 in the first place.

Updating the AGP to version 8.5 should fix this and get rid of those warnings as well:

[task 2024-08-15T16:19:55.760Z] WARNING: We recommend using a newer Android Gradle plugin to use compileSdk = 35
[task 2024-08-15T16:19:55.760Z] 
[task 2024-08-15T16:19:55.760Z] This Android Gradle plugin (8.4.0) was tested up to compileSdk = 34.
[task 2024-08-15T16:19:55.760Z] 
[task 2024-08-15T16:19:55.760Z] You are strongly encouraged to update your project to use a newer
[task 2024-08-15T16:19:55.760Z] Android Gradle plugin that has been tested with compileSdk = 35.
[task 2024-08-15T16:19:55.760Z] 
[task 2024-08-15T16:19:55.760Z] If you are already using the latest version of the Android Gradle plugin,
[task 2024-08-15T16:19:55.760Z] you may need to wait until a newer version with support for compileSdk = 35 is available.
[task 2024-08-15T16:19:55.760Z] 
[task 2024-08-15T16:19:55.760Z] To suppress this warning, add/update
[task 2024-08-15T16:19:55.760Z]     android.suppressUnsupportedCompileSdk=35
[task 2024-08-15T16:19:55.760Z] to this project's gradle.properties.

Can someone else handle this? Otherwise it will have to wait until September when I'm back from vacation.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

I've been poking at updating to AGP 8.5 in my free time, but there's some new issues that I don't have cycles to investigate. I'll get a bug filed for it at least, though.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

I added more jobs to your Try push with AGP 8.6.0, but it appears we're still hitting these failures even with that bumped :(

Comment 19

[Nicolas Guichard]

Indeed, my diagnosis was correct (AGP 8.4 defaults to buildToolsVersion 34 but we only had buildToolsVersion 35 available), but upgrading to 8.6 didn't actually fix that (it still defaults to buildToolsVersion 34).
The proper fix is to be explicit about the buildToolsVersion, things are going green in this Try run: https://treeherder.mozilla.org/jobs?repo=try&revision=456275ef3f9a182262bf3b5c1ef4a84b3b02bbb8 (except a feature-prompts test but that is known to be flaky). I'll push that for review now.

Comment 20

[Nicolas Guichard]

Attachment: Bug 1906365 - Enforce buildToolsVersion in AC/Fenix/Focus. r=nalexander!

This forces Gradle to use the buildToolsVersion we package in the
Android toolchain, and is required for offline building.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd4816137f90
Enforce buildToolsVersion in AC/Fenix/Focus. r=nalexander,android-reviewers,ohall
https://hg.mozilla.org/integration/autoland/rev/b371070ee148
Configure the Glean Gradle Plugin to use our vendored glean_parser. r=nalexander,geckoview-reviewers,android-reviewers,owlish

Comment 22

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/bd4816137f90
https://hg.mozilla.org/mozilla-central/rev/b371070ee148