1523349 - [meta] Teach Searchfox to index and xref between Android Components repositories

Status: RESOLVED DUPLICATE of bug 1530798

(Webtools :: Searchfox, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

Mozilla is developing a huge stack of interconnected Android code for new Android browsers: GeckoView, Android Components, Application Services, the Reference Browser... the list goes on. Only GeckoView is indexed right now, and the Java and Kotlin code are only text-indexed -- but that's better than nothing.

This ticket tracks teaching Searchfox about these new repositories, and hopefully cross-linking between them.

As an instructive example, I was just trying to understand why our various GeckoView-based test beds -- GVE but also r-b -- don't appear to support bare window.open by opening a new tab. Grisha eventually pointed me at https://bugzilla.mozilla.org/show_bug.cgi?id=1510314 and https://github.com/mozilla-mobile/android-components/issues/1503, which might be the correct tickets. But to figure out what was supported I did 10-20 free-text searches across 3 or 4 repositories. If I had more familiarity with Focus, I would have needed another checkout. We can do better!

I know that SF works with git natively, which is good ’cuz all the downstream repos consuming GV are git (and on Github). kats tells me that it's not difficult to get SF to index more repos.

I expect there to be very good indexing tools for Java and less good tools for Kotlin, although I don't need this level of indexing for this to be useful.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Adding more repos and getting text search/blame information is easy enough. We have bug 1490144 on file for adding java/kotlin analysis, so I'm linking that here.

In terms of cross-referencing between repos, we don't yet have ability to do that, and it would be nontrivial to add if we wanted to have all the repos "separate". But if we did something like create a super-repo which just includes the other stuff as sub-repos (the way we currently glue in the mozilla/ folder into the comm-central repo) then that might work better. But that has the downside of requiring extra work to get blame working properly. So either way it's nontrivial but I suspect the super-repo approach will be more user-friendly and probably easier to maintain longer term.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Neat idea: thanks for bringing this up, Nick. I'm excited by this because GitHub's search doesn't seem to match partial words/symbols, making it easy to miss references. Personally, I've been checking out the repository and searching locally with ag.

fwiw, I would very much like to get the code added ASAP and worry later about non-text indexing or cross-references. Just having partial word/symbol matching and search in one location is a great improvement over GitHub.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

https://kats.searchfox.org/mozilla-mobile/source is a WIP that does basic text search if you want to try it out.

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)

https://kats.searchfox.org/mozilla-mobile/source is a WIP that does basic text search if you want to try it out.

kats! kats!!! Amazing work. Can we get https://github.com/mozilla/application-services added to that mix?

Comment 5

[Kartikaya Gupta (email:[e-mail])]

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

Can we get https://github.com/mozilla/application-services added to that mix?

Added. You might need shift+refresh

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

I'm going to mark this fixed by mozilla-mobile by bug 1530798 and bug 1530690.

In terms of this general use case https://github.com/mozilla-mobile/android-components/issues/12751 announced the creation of a mono-repo-ish thing that will live at https://github.com/mozilla-mobile/firefox-android. It seems like this won't completely obsolete the use-case for searchfox's synthetic mozilla-mobile mono-repo, but arguably the right next step for that is likely for the https://github.com/mozilla-mobile org to publish its own mono-repo with automation to keep it updated, ideally using git subtrees (probably?) and searchfox can then index that; this blog post I randomly found via search I think generally captures my thoughts and steps for how that might be done. I believe this would immediately address the lack of annotate/blame support for the existing synthetic mozilla-mobile.