1893181 - Update Searchfox indexing to include AC/Fenix/Focus

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, enhancement, P2)

Description

[Ryan VanderMeulen [:RyanVM]]

I was looking at updating the semanticdb-javac dependency to the latest version and was discussing with asuth how to verify that things work correctly afterwards. During the course of that conversation, it came to light that the Android semanticdb files generated by the indexing job only include GV and related projects but nothing for AC/Fenix/Focus. That seems like something we'd ideally like to improve as a benefit of everything living in one repo?

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

(For clarity, I think all of the changes would happen in m-c as changes to the gradle semanticdb stuff or the packaging logic. Searchfox just unzips the file and runs scip-java on the semanticdb files to build a monolithic java.scip file that will subsequently be processed via scip-indexer into searchfox analysis files.)

Comment 3

[Joe M [:jmahon]]

This would be an important enhancement for Android engineers, since we previously were able to take advantage of this kind of functionality in Github.

Comment 4

[Geoff Brown [:gbrown]]

I think the issue here is that we rely on "mach android compile-all":
https://searchfox.org/mozilla-central/rev/6121b33709dd80979a6806ff59096a561e348ae8/toolkit/mozapps/installer/packager.mk#98
which only runs geckoview tasks:
https://searchfox.org/mozilla-central/rev/55b8c839700564409af5295286a1fd389410e19b/mobile/android/gradle.configure#515-530

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

We have a contractor starting now who can help with this in the short term; is this something you're still interested in pursuing, or is it okay if the contractor takes the bug over?

Comment 6

[Titouan Thibaud [:titouan]]

No problem at all if this person takes this bug over

Comment 7

[Nicolas Guichard]

Attachment: Bug 1893181 - Add Fenix/Focus/Klar to mach android compile-all. r=nalexander!

This extends the mach android compile-all command introduced in
https://phabricator.services.mozilla.com/D192925 to ensure that the
other android targets are built and indexed by Searchfox.

The code is refactored a bit to reduce duplication. It now assumes that
build_config.geckoview == build_config.geckoview_example and that the
same variant is used for Fenix/Focus/Klar.

The searchfox-android task is switched to the online gradle environment
for pip install glean_parser to work as it does not seem to be
available otherwise.

Comment 8

[Nicolas Guichard]

Attachment: Bug 1893181 - Force the Glean Gradle Plugin to use our vendored glean_parser in offline mode. r=nalexander!

This ensures that Android builds that use the offline gradle profile
can install glean_parser.

Comment 9

[Phabricator Automation]

Comment on attachment 9411279 [details]
Bug 1893181 - Force the Glean Gradle Plugin to use our vendored glean_parser in offline mode. r=nalexander!

Revision D215795 was moved to bug 1906365. Setting attachment 9411279 [details] to obsolete.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71eaffe86901
Add Fenix/Focus/Klar to mach android compile-all. r=nalexander,geckoview-reviewers,owlish

Comment 11

[Iulian Moraru]

Backed out 2 changesets (bug 1906365, bug 1893181) for causing mass android-all build bustages.

• Push with failures

• Failure log

• Backout link

[task 2024-08-15T16:15:31.933Z] To suppress this warning, add/update
[task 2024-08-15T16:15:31.933Z]     android.suppressUnsupportedCompileSdk=35
[task 2024-08-15T16:15:31.933Z] to this project's gradle.properties.
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] FAILURE: Build failed with an exception.
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] * What went wrong:
[task 2024-08-15T16:15:32.233Z] Could not determine the dependencies of task ':support-test:extractDebugAnnotations'.
[task 2024-08-15T16:15:32.233Z] > Failed to find Build Tools revision 34.0.0
[task 2024-08-15T16:15:32.233Z] 
[task 2024-08-15T16:15:32.233Z] * Try:
[task 2024-08-15T16:15:32.233Z] > Run with --stacktrace option to get the stack trace.
[task 2024-08-15T16:15:32.233Z] > Run with --info or --debug option to get more log output.
[task 2024-08-15T16:15:32.234Z] > Run with --scan to get full insights.
[task 2024-08-15T16:15:32.234Z] > Get more help at https://help.gradle.org.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
[task 2024-08-15T16:15:32.234Z] 
[task 2024-08-15T16:15:32.234Z] BUILD FAILED in 1m 27s
[task 2024-08-15T16:15:32.234Z] 10 actionable tasks: 10 executed
[taskcluster 2024-08-15 16:15:33.105Z] === Task Finished ===

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Bug 1906365 was the culprit.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3afe0a983f1a
Add Fenix/Focus/Klar to mach android compile-all. r=nalexander,geckoview-reviewers,owlish

Comment 14

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/3afe0a983f1a

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Can we verify that the indexing is now working as expected? I did a quick sampling of a couple different Fenix .kt files but don't see a change in presentation from before this landed.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Ugh, from today's run:

[task 2024-09-11T11:08:42.670Z] 11:08:42     INFO -  package> FAILURE: Build failed with an exception.
[task 2024-09-11T11:08:42.670Z] 11:08:42     INFO -  package>
[task 2024-09-11T11:08:42.670Z] 11:08:42     INFO -  package> * What went wrong:
[task 2024-09-11T11:08:42.671Z] 11:08:42     INFO -  package> Could not determine the dependencies of task ':browser-domains:compileDebugJavaWithJavac'.
[task 2024-09-11T11:08:42.671Z] 11:08:42     INFO -  package> > Failed to find Build Tools revision 34.0.0

https://firefoxci.taskcluster-artifacts.net/H527cpE2T8227-hQ4aPnBg/0/public/logs/live_backing.log

Comment 17

[Nicolas Guichard]

It seems the changes I made to enforce buildToolsVersion for AC/Fenix/Focus didn't cover the case of building from the toplevel Gradle project, and I didn't notice because mach android compile-all didn't fail when the Gradle build failed.

I'll send patches for review if that Try run passes.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

nvm

Comment 19

[Nicolas Guichard]

Yes the second commit in that Try run should cover that for mach android compile-all.
I see there are however a couple of other commands in android/mach_commands.py like build-geckoview_example that unconditionally return 0.

Comment 20

[Nicolas Guichard]

Attachment: Bug 1893181 - Enforce buildToolsVersion from toplevel Gradle project too. r=nalexander!

The resolution of bug 1906365 was incomplete, the buildToolsVersion was
not enforced when building AC, Fenix or Focus from the toplevel Gradle
project.

Comment 21

[Nicolas Guichard]

Attachment: Bug 1893181 - Make `mach android compile-all` fail when Gradle build fails. r=nalexander!

The issue fixed by parent commit was not caught by the CI because
mach android compile-all didn't fail when the Gradle build failed.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d731e2204fc6
Enforce buildToolsVersion from toplevel Gradle project too. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/2f81634f5419
Make `mach android compile-all` fail when Gradle build fails. r=nalexander,geckoview-reviewers,ohall

Comment 23

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/d731e2204fc6
https://hg.mozilla.org/mozilla-central/rev/2f81634f5419

Comment 24

[Andrew Sutherland [:asuth] (he/him)]

This seems like it's working great, wooo!

I think I need to add ontology mappings to teach searchfox about Java runnables and the Android ListenableWorker thing. Like the diagram calls-to:'org::mozilla::fenix::utils::NotificationBaseKt::createBaseNotification' depth:4 is neat, but it hits a wall because of this newfangled async programming, it seems like.

I'll also see if there's an easy way to provide some glue for the GeckoThread. Because of the recent ContentChild weird crash bug 1915788 I wanted to see if calls-to:NS_InitXPCOM could get into fenix, but because there's a static INSTANCE we lose the trail. A search on re:private static final .+ INSTANCE finds no false positives for a singleton idiom, so maybe a rule on fields named INSTANCE with a type_pretty prefix of private static final could work[1].

1: GeckoThread's structured JSON is:

{"loc":"00135:35-43","structured":1,"pretty":"org::mozilla::gecko::GeckoThread::INSTANCE","sym":"S_jvm_org/mozilla/gecko/GeckoThread#INSTANCE.","type_pretty":"private static final GeckoThread INSTANCE","kind":"field","subsystem":null,"parentsym":"S_jvm_org/mozilla/gecko/GeckoThread#","implKind":"impl","sizeBytes":null,"ownVFPtrBytes":null,"bindingSlots":[],"ontologySlots":[],"supers":[],"methods":[],"fields":[],"overrides":[],"props":[],"variants":[]}

which doesn't provide a ton of extra metadata to key on versus what we get from our clang indexer (which makes sense).