Closed Bug 1942757 Opened 26 days ago Closed 19 days ago

AppConstants.sys.mjs is not indexed

Categories

(Webtools :: Searchfox, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: arai, Assigned: arai)

References

Details

Attachments

(3 files)

bug 1937699 added preprocessed files to the generated files list, and the preprocessed Readerable.sys.mjs is now indexed, but the preprocessed AppConstants.sys.mjs is not indexed for some reason.
Perhaps it's caused by the content being different between architecture?

(In reply to Tooru Fujisawa [:arai] from comment #0)

Perhaps it's caused by the content being different between architecture?

This is probably related, yeah:

[2025-01-20T17:05:54Z INFO  tools::file_format::analysis] Error trying to open analysis file [analysis-*/__GENERATED__/./dist/bin/update-settings.ini]
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from linux64
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from macosx64
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from macosx64-aarch64
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from win64
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from android-aarch64
Taking generated file ./dist/bin/modules/AppConstants.sys.mjs from ios
Generated file ./dist/bin/modules/Readerable.sys.mjs was identical across platforms where it was created

And doing a find in m-c:

ubuntu@ip-172-31-28-166:~/index/mozilla-central$ find . -name AppConstants.sys.mjs\*
./git/toolkit/modules/AppConstants.sys.mjs
./description/toolkit/modules/AppConstants.sys.mjs
./analysis/toolkit/modules/AppConstants.sys.mjs.gz
./analysis/toolkit/modules/AppConstants.sys.mjs
./file/toolkit/modules/AppConstants.sys.mjs.gz
./file/toolkit/modules/AppConstants.sys.mjs
./detailed-per-file-info/toolkit/modules/AppConstants.sys.mjs
./objdir/__ios__/dist/bin/modules/AppConstants.sys.mjs
./objdir/__macosx64-aarch64__/dist/bin/modules/AppConstants.sys.mjs
./objdir/__android-aarch64__/dist/bin/modules/AppConstants.sys.mjs
./objdir/__linux64__/dist/bin/modules/AppConstants.sys.mjs
./objdir/__macosx64__/dist/bin/modules/AppConstants.sys.mjs
./objdir/__win64__/dist/bin/modules/AppConstants.sys.mjs

And then checking our files listings, this may not be lined up correctly:

ubuntu@ip-172-31-28-166:~/index/mozilla-central$ grep AppConstants.sys.mjs *.list
generated-files-android-aarch64.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files-ios.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files-linux64.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files-macosx64-aarch64.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files-macosx64.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files-win64.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs
generated-files.list:./dist/bin/modules/AppConstants.sys.mjs

Looks like there's dependency between scripts that's not handled properly.

The objdir-dir and objdir-files are generated from the list of files in analysis, which is created by find-objdir-files.sh,
but at that point, the analysis for JS files aren't yet created, given they're generated by js-analyze.sh, which is done after find-objdir-files.sh.

So, the directory file/__GENERATED__/__linux64__/dist/bin is not created by objdir-mkdirs.sh.

I guess, we can reorder some of them.

$MOZSEARCH_PATH/scripts/find-objdir-files.sh
$MOZSEARCH_PATH/scripts/objdir-mkdirs.sh

date

URL_MAP_PATH=$INDEX_ROOT/aliases/url-map.json
DOC_TREES_MAP=$INDEX_ROOT/doc-trees.json

$MOZSEARCH_PATH/scripts/process-chrome-map.py $GIT_ROOT $URL_MAP_PATH $INDEX_ROOT/*.chrome-map.json\
 || handle_tree_error "process-chrome-map.py"

date

$MOZSEARCH_PATH/scripts/js-analyze.sh $CONFIG_FILE $TREE_NAME || handle_tree_error "js-analyze.sh"

The script that depends on objdir-* seems to be only ipdl-analyze.sh.
So maybe we can move the find-objdir-files.sh+objdir-mkdirs.sh immediately before that, or maybe create a temporary list and then re-run those scripts to generate the final list.
I'll see how it goes locally.

Also realized that the generated files are not processed by other analysis, such as js-analyze.sh.
That part also needs to be solved if we rely on the existence of analysis files.

The other option would be to directly reflect the files in objdir, instead of using analysis files.

(In reply to Tooru Fujisawa [:arai] from comment #4)

The other option would be to directly reflect the files in objdir, instead of using analysis files.

If this simplifies things, this seems good. We do already prioritize the file itself instead of the analysis file in the merge process, so improving consistency around that makes sense, especially if it's causing these problems.

Attached file new-files-and-dirs.txt

Using objdir file list looks working.
At least there's no missing files/dirs in the list.
There are some newly added files/dirs (see the attachment).

I'll test some more (and fix the test repo around the expecation or the intermediate files) and post PR.

Neat, those all seem helpful. The __RUST_BUILD_SCRIPT__ paths in particular are probably quite nice to have!

The only thing that seems like it might not be desired is __GENERATED__/rust-analyzed but that's correctly there because we intentionally put it there but we no longer should be putting it there because the change to use rust-analyzer and SCIP and that we don't automatically run rust-analyzer means that we don't need a marker file to tell us not to automatically run it. That is, we should remove those lines that touch the file.

The patch to use the actual files in objdir is now available on https://arai.searchfox.org/

The preprocessed AppConstants.sys.mjs files are shown in the search, but not yet analyzed. I'll handle it in separate bug.
https://arai.searchfox.org/mozilla-central/search?q=&path=AppConstants.sys.mjs&case=false&regexp=false

The rust bindings are also shown (also not analyzed yet)
https://arai.searchfox.org/mozilla-central/source/__GENERATED__/__linux64__/__RUST_BUILD_SCRIPT__/gecko-profiler/gecko/bindings.rs

Attached file GitHub Pull Request
Assignee: nobody → arai.unmht
Attached file GitHub Pull Request

for the rust-analyzed file part in the mozsearch-mozilla repo

Status: NEW → RESOLVED
Closed: 19 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: