1432475 - Generate rust analysis data in searchfox taskcluster job

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

In order to switch searchfox from building its own analysis data to using the analysis data from the taskcluster job (bug 1425597) we need to make sure that the taskcluster job is actually building everything we need. Recently searchfox added rust analysis support, and so we need to add that to the taskcluster job as well.

Note that this requires switching the taskcluster job from rust-stable to rust-nightly since only rust nightly supports the -Zsave-analysis flag which we need. This does mean that it's more liable to break, but it will be no worse than what searchfox is doing now.

Here's a try push where I attempted this yesterday: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbb4da4c06c64e98ea2b8fe4b169157eca95982f - it failed but because of a nightly-only build error that is being addressed. I'll try again once that's fixed.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=85eacf7bf46f24a6bd0fe44fb0bf50d614d54c80 - still failing, different error this time. I suspect this is rust nightly compiler bustage since the code that's failing to build hasn't changed recently and builds fine with rust stable.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

It wasn't really nightly compiler bustage. The code that was errorring out was the slog crate, which got pulled in as part of the geckodriver crate's dependency tree. That only happens because on taskcluster we're running the build with MOZ_AUTOMATION=1, which doesn't happen in searchfox's own build. But indexing geckodriver is a good thing, so I tried updating slog to the newest compatible version, and that seemed to fix the problem: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc76d881d8ddfe2855b0976eeb5169b319a746c5

I'll put up patches for this later.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

The above try push missed adding the new zipfile to the uploaded artifacts. Here's one with that fixed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ac5fde22e60ebaf7f65b96c246c2362c10455294

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1432475 - Bump slog to a version that compiles with Rust nightly.

Review commit: https://reviewboard.mozilla.org/r/215344/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215344/

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

This requires running the mozsearch job with a nightly rust.

Review commit: https://reviewboard.mozilla.org/r/215346/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215346/

Comment 6

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8945115 [details]
Bug 1432475 - Bump slog to a version that compiles with Rust nightly.

https://reviewboard.mozilla.org/r/215344/#review220922

If I would’ve thought bumping slog also would mean bumping the other
slog-* dependencies.  In any case, if this builds fine on TC then
this is good.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Yeah it seems to build fine. The actual code changes between the two versions were minor, so that's probably why it's still compatible with the other slog-* crates.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945116 [details]
Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

https://reviewboard.mozilla.org/r/215346/#review220964

I'll defer to :glandium here -- I see nothing that concerns me!

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 8945116 [details]
Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

https://reviewboard.mozilla.org/r/215346/#review222258

::: taskcluster/ci/toolchain/linux.yml:342
(Diff revision 1)
> +    description: "rust repack"
> +    treeherder:
> +        kind: build
> +        platform: toolchains/opt
> +        symbol: TL(rust-nightly)
> +        tier: 1

Is the searchfox job tier 1?

::: taskcluster/ci/toolchain/linux.yml:352
(Diff revision 1)
> +            UPLOAD_DIR: artifacts
> +    run:
> +        using: toolchain-script
> +        script: repack_rust.py
> +        arguments: [
> +            '--channel', 'nightly',

This is rather undesirable, because the version you get depends on when this job runs. You should use a dated nightly. Or no nightly at all, if -Zsave-analysis can still work if you set RUSTC_BOOTSTRAP=1.

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8945115 [details]
Bug 1432475 - Bump slog to a version that compiles with Rust nightly.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215344/diff/1-2/

Comment 11

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8945116 [details]
Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215346/diff/1-2/

Comment 12

[Kartikaya Gupta (email:[e-mail])]

(In reply to Mike Hommey [:glandium] from comment #9)
> 
> Is the searchfox job tier 1?

Good catch, I downgraded this to tier 2

> > +            '--channel', 'nightly',
> 
> This is rather undesirable, because the version you get depends on when this
> job runs. You should use a dated nightly. Or no nightly at all, if
> -Zsave-analysis can still work if you set RUSTC_BOOTSTRAP=1.

Ok, I've changed it to use a dated nightly. I don't know what RUSTC_BOOTSTRAP does and searching for it didn't provide any info that seemed relevant to this problem.

Try push with the updated patches: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6e2c4814f6a2530547cc68f021930323d43fd8f6

Comment 13

[Mike Hommey [:glandium]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #12)
> (In reply to Mike Hommey [:glandium] from comment #9)
> > 
> > Is the searchfox job tier 1?
> 
> Good catch, I downgraded this to tier 2
> 
> > > +            '--channel', 'nightly',
> > 
> > This is rather undesirable, because the version you get depends on when this
> > job runs. You should use a dated nightly. Or no nightly at all, if
> > -Zsave-analysis can still work if you set RUSTC_BOOTSTRAP=1.
> 
> Ok, I've changed it to use a dated nightly. I don't know what
> RUSTC_BOOTSTRAP does and searching for it didn't provide any info that
> seemed relevant to this problem.

It enables nightly features for when rustc bootstraps itself. We abuse it for simd. We could do the same here if it works.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8945116 [details]
Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215346/diff/2-3/

Comment 15

[Kartikaya Gupta (email:[e-mail])]

It does work, thanks for the tip! Patch updated accordingly, and since we don't need nightly rust I dropped the first patch too.

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 8945116 [details]
Bug 1432475 - Enable rust analysis on mozsearch jobs and save the output to a build artifact.

https://reviewboard.mozilla.org/r/215346/#review222578

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e90c77e89c5
Enable rust analysis on mozsearch jobs and save the output to a build artifact. r=glandium

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/6e90c77e89c5