Closed Bug 1425597 Opened 8 years ago Closed 7 years ago

Use the mozilla-central searchfox index generated by taskcluster cron jobs

Categories

(Webtools :: Searchfox, defect)

Product:
Webtools
Component:
Searchfox
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kats, Assigned: kats)

References

Details

Now that the searchfox indexing job is running as a taskcluster cron job on mozilla-central, the most recently built index can be found at this URL: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-searchfox-debug/artifacts/public/build/target.mozsearch-index.zip We should update mozsearch to pull this zip file and use the contents rather than running the clang-plugin directly. Presumably the way to approach this is to allow the config.json file to specify the URL and then have the scripts use the URL if available.
I guess one thing we need to keep in mind is that the version of code for which the index data was generated might be different from the version of code which we're using to generate the static pages. Probably best to record the version for which the index was generated and use that version for the static pages too.
I have some partial patches for this. I should just post what I have. I'll try to remember later today.
I started hacking on this. Totally untested WIP is at https://github.com/staktrace/mozsearch-mozilla/commit/09d6a565d53902674d42851db89db64aad3dd1ef
Assignee: nobody → bugmail
So the first problem with the above patch is that it looks like |mach cargo check| doesn't work on an unbuilt tree. I might be able to just a |mach configure| and then run cargo check. Another option is to modify the taskcluster job to also save the rust analysis files, but then I realized that generating the rust analysis requires rust nightly, and on taskcluster we're building with rust stable (or whatever gecko is currently building with). So that might be non-trivial, but I'll give it a shot.
Quick update here: the taskcluster job now generates the rust analysis files, but we need to update the analyzer to deal with the paths being wrong. Since the taskcluster jobs build stuff in /builds/worker/workspace/build/src/... that's what the save-analysis files have, and then when we run rust-indexer.rs it can't find those files. That should be pretty easy to fix up. Another problem is that the generated-files tarball from the taskcluster build doesn't actually contain all the generated sources. We do have analysis data for those generated sources, though, and that results in a "mismatch" where we try and create rendered versions of those source files but fail because we don't have the generated source. I filed bug 1440879 to try and get that resolved somehow. The final problem I have right now is slightly more mundane - because we get analysis data for a particular revision from taskcluster, we have to make sure the local gecko-dev and gecko-blame repos on the indexer instance are synced to the same revision. I have code that does this, by checking out gecko-dev to the equivalent git revision and them building the blame only up to that revision. However this doesn't handle the case where the blame has already been built to a newer revision, because then the blame repo will be at a newer revision than what we want. We don't check out the blame repo when using it, so I have to find the code in output-file.rs where we use the head of the blame repo and modify it so we use the specific revision that corresponds to the source. This is sort of a one-time problem in that it would only occur once during the migration from non-taskcluster to taskcluster (because in all other cases we should be monotonically advancing with respect to the revision being used), but it's still good to fix this in a robust manner. The current patch is at https://github.com/staktrace/mozsearch-mozilla/commit/4f3b54244a2fce47415de42472021cc3006ee0d7 - so far I haven't needed to make any changes to the mozsearch repo for this.
Ran into another problem with the rust analysis parsing: https://github.com/nrc/rls-analysis/issues/130
Status update: it's impractical to make rls-analysis support save-analysis files from different versions of rustc, so I believe the best long-term solution here is to run the rls-analysis step as part of the taskcluster job. This effectively means running rust-indexer.rs as a post-build step on m-c, and then downloading the produced index files to the searchfox indexer run, rather than using the raw save-analysis files. If we then add the same indexing job to e.g. mozilla-beta which is using a different version of rustc and producing different save-analysis files, we will still be fine, because the rust-indexer.rs on mozilla-beta will be compatible with that, and the produced index files will still be something the searchfox indexer can work with. That being said, doing that step is a blocker for bug 1282123, not this bug. We should be able to finish off this bug as long as we make sure the version of rls-analysis we are using on the searchfox side is compatible with whatever version of rustc is being used to build m-c. I'll see if I can push this over the line.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) (parental leave) from comment #5) > Quick update here: the taskcluster job now generates the rust analysis > files, but we need to update the analyzer to deal with the paths being > wrong. Since the taskcluster jobs build stuff in > /builds/worker/workspace/build/src/... that's what the save-analysis files > have, and then when we run rust-indexer.rs it can't find those files. That > should be pretty easy to fix up. This turns out to not actually be a problem. The save-analysis files only uses the full path (with the /builds/worker/... thing) for external crate references, which we don't use in rust-indexer.rs anyway. For everything else the save-analysis files are relative to the src tree, so it Just Works when we run it through rust-indexer.rs. > Another problem is that the generated-files tarball from the taskcluster > build doesn't actually contain all the generated sources. We do have > analysis data for those generated sources, though, and that results in a > "mismatch" where we try and create rendered versions of those source files > but fail because we don't have the generated source. I filed bug 1440879 to > try and get that resolved somehow. For now I resolved by deleting the analysis data for any generated file for which we don't have the generated file itself. This is not great but ok for now. > The final problem I have right now is slightly more mundane - because we get > analysis data for a particular revision from taskcluster, we have to make > sure the local gecko-dev and gecko-blame repos on the indexer instance are > synced to the same revision. I have code that does this, by checking out > gecko-dev to the equivalent git revision and them building the blame only up > to that revision. However this doesn't handle the case where the blame has > already been built to a newer revision, because then the blame repo will be > at a newer revision than what we want. We don't check out the blame repo > when using it, so I have to find the code in output-file.rs where we use the > head of the blame repo and modify it so we use the specific revision that > corresponds to the source. This is sort of a one-time problem in that it > would only occur once during the migration from non-taskcluster to > taskcluster (because in all other cases we should be monotonically advancing > with respect to the revision being used), but it's still good to fix this in > a robust manner. Fixed this, I just needed to do a `git reset --soft <rev>` on the blame repo to make it use the appropriate revision. Latest patch is at https://github.com/staktrace/mozsearch-mozilla/commit/3650c7103dd6e763c0e44b9f60fdefc7f480b9ba, and the current instance on dev.searchfox.org is built using this. It seems to work ok. I'll want to run it a few more times on different versions of m-c before declaring it fit for use. In the meantime I can look into bug 1440879 to get those missing generated files.
Dependencies are fixed, so there's nothing blocking this now. I'll do some more final testing before pushing the PR for review.
This is deployed now.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.