1761287 - Switch to rust-analyzer's LSIF or SCIP output representations from save-analysis for rust indexing since -Zsave-analysis is going away

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

:khuey indicated[1] that rust-analyzer's LSIF output (rust-analyzer PR 10181 landed late Sep, 2021) is very usable, so similar to the idea of bug 1740290, we should investigate switching over to ingesting LSIF for rust. There could of course be some synergies with JS/TS LSIF ingestion.

The main thing I want to understand right now about LSIF is where the semantic mapping layer is to symbols, etc. In the limited skimming of LSIF that I've done previously, LSIF was explicitly focused on operating in text-span space because the primary consumer has always been text editors. There's a graph structure though, so ideally the span edges map directly to usable symbol namespaces which searchfox can then normalize on ingestion.

1: cool screenshot at https://twitter.com/khuey_/status/1506401296575655936

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

With the sunsetting of RLS in https://github.com/rust-lang/rust/pull/100863 discussion there and more specifically on the -Zsave-analysis tracking issue at https://github.com/rust-lang/rust/issues/43606 it sounds like -Zsave-analysis will be going away because there's no longer a need for it anymore. Searchfox is believed to be the only meaningful consumer which is at once very glamorous and also makes it clear save-analysis can't stick around just for searchfox' benefit.

This means there is potentially some urgency for a migration to LSIF or SCIP as rust language indexing will break otherwise.

In terms of my comments in comment 0, I did dig into the LSIF rep and the "moniker" mechanism in general seems like it will tend to provide the glue we need. That said, it seems like we'd always want to favor the SCIP representation which landed in rust-analyzer 6 days ago in https://github.com/rust-lang/rust-analyzer/pull/12976 over LSIF because searchfox very much likes having the additional semantics available.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

There is now a patch to remove the save-analysis support at https://github.com/rust-lang/rust/pull/101841 which would break our rust indexing, although there is no timeline for landing as of yet and there is a specific request to delay for 2-3 weeks.

Note that the impact of that will be buffered by how we build with rust on mozilla-central. Specifically, as expressed in toolchains.yml:

# We build stable rust from source so the resulting compiler acts as a nightly
# rust compiler, allowing to use unstable features like -Zbuild-std and
# sanitizers.
rust-1.63.0:
    description: Rust 1.63.0 source code
    fetch:
        type: git
        include-dot-git: true
        repo: https://github.com/rust-lang/rust/
        revision: 4b91a6ea7258a947e59c6522cd5898e7c0a6a88f

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Attachment: PR: Emilio's WIP

Emilio has a WIP at https://github.com/mozsearch/mozsearch/pull/570

Comment 4

[Mike Hommey [:glandium]]

cf. bug 1829444, it happened, save-analysis is gone. Do we need to keep searchfox on 1.68 or are we ready to move on?

Comment 5

[Emilio Cobos Álvarez (:emilio)]

Hah, just yesterday I wrote https://github.com/mozsearch/mozsearch/pull/626, and in fact I have a scip run up in https://dev.searchfox.org/

If you just remove the save-analysis flag as long as we have a scip index we should be good to go. There are some regressions:

• The first one (no stdlib symbols) is due to our custom rust stdlib stuff which we can remove too, so should be easy to address.
• The second one is very minor and I don't think it's a huge deal either way.
• The third one is a bit harder to address, but it's arguably not a huge deal either, and we could add heuristics to the indexer if we wanted to to have a short-term (even if not general) solution.

So I'd say we can move on from save-analysis, and I can work on those regressions in the coming weeks. How does that sound?

Comment 6

[Mike Hommey [:glandium]]

Sounds good. What do you want to do with the immediate problem? Remove the save-analysis flag or temporary go back to 1.68 for the searchfox jobs?

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Let's try to remove the save-analysis flag first.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Emilio's awesome work on this landed some time ago now! Woo!