1847632 - Style crate's rust build byproduct "gecko/struct.rs" that should be present under `__RUST_BUILD_SCRIPT__/style/gecko/structs.rs` is not present in the index, but is present in the indexing objdirs. Generated files may need analysis files to work.

Status: NEW

(Webtools :: Searchfox, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

I noticed that "gecko/structs.rs" is not present in searchfox's index; it really should be. There are some potentially distinct factors here from bug 1846041 although I think there may also be some overlap.

Investigation (Verbose)

I went looking and found that it is available to the indexer. Here's a truncated find . -name structs.rs invocation from the current config1 spare:

./git/third_party/rust/dns-parser/src/structs.rs
./git/third_party/rust/mach2/src/structs.rs
./git/third_party/rust/mach/src/structs.rs
./objdir/__win64__/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
./objdir/__linux64__/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
./objdir/__android-armv7__/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
./objdir/__macosx64__/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
./file/third_party/rust/dns-parser/src/structs.rs
./file/third_party/rust/mach2/src/structs.rs
./file/third_party/rust/mach/src/structs.rs

But some other files that are identified in https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/backend/cargo_build_defs.py like atom_macros.rs are present, although that example is explicitly generated by a python script whereas structs.rs is generated by "rust-bindgen".

A find invocation for "atom_macro.rs" gets us:

./detailed-per-file-info/__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
./analysis/__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
./objdir/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
./file/__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs

Grepping for the relative path "gecko/structs.rs" in our files list, we do find:

generated-files-android-armv7.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files-linux64.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files-macosx64.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files-win64.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/structs.rs

Comparing that with atom_macro.rs we can notice the file is not listed in all-files or objdir-files, whoops!

generated-files-android-armv7.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files-linux64.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files-macosx64.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files-win64.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
generated-files.list:./__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
all-files:__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs
objdir-files:__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/atom_macro.rs

We can identify the origin of objdir-files as find-objdir-files.sh where we can see it's deriving the list from a find command invoked in ${INDEX_ROOT}/analysis/__GENERATED__.

Looking at the same old local log from above from April that was using rust-indexer, on line 31944 we see the following interesting output that may be relevant even though we no longer actually emit this because this was from save-analysis processing:

[INFO rust_indexer] Skipping nonexistent source file with searchfox path '__GENERATED__/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs' which mapped to local path '/home/vagrant/mozilla-index/mozilla-central/git/__RUST_BUILD_SCRIPT__/style/gecko/structs.rs'

While the above potentially suggests some problems with our order of operations and/or the paths we provide to rust-indexer or scip-indexer, bug 1846041 may be involved. In my hobby branch I disabled the rust checks for GeneratedType because rust-analyzer as invoked was not analyzing the generated.rs created by the test repo's build.rs.

Problem Identification

It seems like accordingly there might be 2 or 3 problems here:

1. We have no analysis data for gecko/structs.rs and this is due to some combination of rust-analyzer limitations or insufficient configuration or misconfiguration on our part.
2. It seems like we require the presence of analysis data for generated files in order for them to show up. We might just want to "touch" empty analysis files into existence for generated files that we have locally but which we didn't end up producing analysis files for somehow.
3. Something may be wrong with the platform merging step in terms of when we propagate the source files or how we point rust-indexer/scip-indexer at generated source files (although both binaries diverge in how they figure those things out). It's not particularly hard to just have the binaries try multiple path roots if appropriate.