958852 - [searchfox-Rust] Index the Rust repo

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Nick Cameron [:nrc]]

Being able to index the Rust repo (librustc, etc.) is the first goal for Rust DXR support.

Comment 1

[Nick Cameron [:nrc]]

I managed to get all the way through the index today (it was tough going to get there - I had to fix a bunch of Rust bugs along the way). The log contains about 12,000 indexing errors and there were two overlapping spans. My impression of the indexing was that (excluding enums and type/lifetime params) we are about 75% there.

I'm not sure if my strategy for dealing with spans and sub-spans is robust enough - there were a lot of errors due to that. It is extremely frustrating that rustc doesn't keep around more spans.

pelmers' work with enums should knock down a bunch of those errors and there are a bunch more I think I know how to fix.

Comment 2

[Erik Rose [:erik][:erikrose]]

Wow, that sounds like great progress. I assume you mean overlapping <a> tags; overlapping spans are expressly supported. I'm not sure what we do about overlapping spans in the JS atm; we might have to improve it to not stop looking for context-menu data at the innermost one. Did you have to do much to the DB schema to support Rust?

Comment 3

[Nick Cameron [:nrc]]

(In reply to Erik Rose [:erik][:erikrose] from comment #2)
> Wow, that sounds like great progress. I assume you mean overlapping <a>
> tags; overlapping spans are expressly supported. I'm not sure what we do
> about overlapping spans in the JS atm; we might have to improve it to not
> stop looking for context-menu data at the innermost one. Did you have to do
> much to the DB schema to support Rust?

Yeah, it's a bit of a milestone, although it also shows there's a long way to go. Yes, I meant overlapping <a> tags. To the language-neutral part of the schema I made a few additions, nothing breaking. And added a whole bunch of plugin-specific tables. Again, none of them overlap with Clang.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

We are not going to work on that!

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Let's save a bug by reopening it :)

Comment 7

[Kartikaya Gupta (email:[e-mail])]

So the rust repo has a bunch of submodules including a complete fork of llvm which makes things a bit tricky. Either we can initialize the submodules and not have blame, or we can skip the submodules and maybe have blame info. Right now we don't support blame for repos with submodules. In theory that's fixable but will take some work.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

The repo with submodules checked out is 3.4G which is pretty large. I guess I'd lean towards no submodules initially unless they're needed.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

I built the git and blame tarballs and uploaded them to S3. But the indexer run fails because we ask codesearch to walk submodules and the submodules don't get checked out. So we probably need a repo config option to disable walk_submodules.

Comment 10

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch/pull/383
https://github.com/mozsearch/mozsearch-mozilla/pull/118

Comment 11

[Kartikaya Gupta (email:[e-mail])]

PRs merged. I also updated the release-lb load balance to redirect /rust/* to the release4 web-server and triggered a new run of that indexer. Should be done pretty soon. And then when the daily release1 indexer is up it should have link to the repo on the main page as well.