Closed Bug 1516568 Opened 5 years ago Closed 5 years ago

Stand up webrender CI on Firefox CI (macOS edition)

Categories

(Core :: Graphics: WebRender, enhancement, P2)

Product:
Core
Component:
Graphics: WebRender
Version:
Other Branch
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox67 --- fixed

People

(Reporter: kats, Assigned: kats)

References

(Depends on 1 open bug)

Details

(Whiteboard: [gfx-noted])

Attachments

(7 files)

Bug 1516568 - Allow run-task jobs on docker-worker to use internal tooltool downloads. r?ahal
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Update osmesa-src to includes servo/osmesa-src#33 for cross-compiling support. r?kvark
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Update WR mac CI scripts to allow cross-compiling. r?kvark
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Fuzz a couple of mac reftests. r?gw
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Add jobs to cross-compile wrench and use the resulting binaries. r?glandium,jrmuizel
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Add jobs to cross-compile cargo test binaries and run them on macOS. r?glandium,jrmuizel
47 bytes, text/x-phabricator-request
Details | Review
Bug 1516568 - Provide path to python executable when invoking fetch-content script. r?ahal
47 bytes, text/x-phabricator-request
Details | Review 
+++ This bug was initially created as a clone of Bug #1507884 +++

Same deal as bug 1507884, but for macos
On the github-taskcluster workers we manually installed pkg-config, zlib, and cmake via brew (which also needed to be installed). We'll have to ensure those are available here somehow.
Is it easy for us to build the artifacts on the linux cross builders like we do for Gecko builds?
Flags: needinfo?(kats)
It might be possible, but I'm not sure it would be particularly easy or clean. To run wrench on macos we need the osmesa libs in the right spots [1] and I'm not sure if there are recursive dependencies we'd need to expose as artifacts and then copy over to the job running the reftests.

Currently I'm exploring downloading the osmesa deps the way we do the wrench vendoring on linux - i.e. have a job that installs brew, installs the brew things we need, and then export that as an artifact. I'll see how that goes first.

[1] https://searchfox.org/mozilla-central/rev/8a135a9c5a96b59269f544fcaee76d8fd5a7026a/gfx/wr/wrench/script/headless.py#60-65
Flags: needinfo?(kats)
ISTR us deciding to not stand up these jobs on mac when we discussed it a few weeks ago? No strong objection, just looking to reduce work here.
Agreed that it doesn't need to block. I just wanted to see if I could get a sense of how much work it would be. The approach I described in comment 4 has some promise but I ran into some chain-of-trust problem with generic-worker on macOS that I don't know how to resolve. I'll file a bug for that and put this on hold for now.

Tried this again just using a zlib in a tarball that I checked in as part of my commit (to eventually go into tooltool). But rustc is blowing it's stack trying to compile gleam, it looks like.

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221503568&repo=try&lineNumber=2863

Nightly build from October (what we use in other places in Firefox CI) has the same gleam error.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=62162a0ba885e7979872bbe40d9250cc54db8d85

More recent Nightly produces unused variable warning which appears wrong.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cade13ebd3e03c38386512a5ddc5735e9d359107

Sigh. I'll try the cross compilation approach.

I did try the cross compilation approach (many try pushes at https://treeherder.mozilla.org/#/jobs?repo=try&author=kgupta%40mozilla.com&group_state=expanded&fromchange=8e20207ab2a48a080a6714ffb53b7d080dd3cffe&tochange=3fc5d6e18e26a5a0b0729e291a17f086af6a4f37) but wasn't able to get it working. Mostly I don't know what the magical incantation is that will let me do a "./configure && make" type build using clang with cross-compilation. Traditionally with gcc you just have a separate toolchain for cross-compilation and you point your CC/CXX variables to that, but with clang it's all in the same binary and you need to pass flags. I may or may not have gotten that working but then I ran into linking problems that I couldn't make sense of.

I made some progress on the cross-compilation approach in that I now have osmesa-src cross-compiling locally although I didn't run the stuff on a mac yet.

And for the record I'm still getting bizarre errors with recent Nightly rust for macOS. I'm suspecting the rust-repack script doesn't do a good job in making a work rustc when I specify host and target to be macOS. I'll stick with the cross-compilation for now since that approach is showing more promise.

For posterity https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=e07e1713a4a662c0e25b2133e076d715dce0f42d is the try push where I have wrench cross-compiling on a linux builder for mac, and then a mac task that uses the cross-compiled wrench to run the reftests. It runs and there's fuzziness on two mac-only reftests, hence the job failure.

For the "debug" job I was attempting to run the macos-debug-tests script on a Linux builder with the cross-compiling setup but the cargo test commands also try to execute the produced test binaries so that's not going to work. I'll have to ship the test binaries over to a mac worker for running. (Or figure out how to get proper mac builds going).

Also jdm pointed me to https://github.com/servo/gleam/issues/180 which is the problem I was seeing in comment 7.

Now with the cargo test binaries cross compiled and run on a macOS worker: https://treeherder.mozilla.org/#/jobs?repo=try&revision=889b84915ba01c0fd6f4ad90f1bc51297d1d5930

I had to fiddle with CC/CXX/CFLAGS/CXXFLAGS a bit more to make both mozangle and osmesa-src happy. Will split up the patches and put them up for review tomorrow.

This code already exists for other (non-run-task jobs) on docker worker,
so this patch just reuses the existing code but adds a bit to the
run-task schema to allow the task to opt in.

This introduces some env vars to allow running cross-compiled
binaries instead of running things on the builder. Additionally
the cargo test --features "ipc" check is modified to be check
instead since there are no actual tests being run by that command.
The only thing we lose is a rustdoc example check but we are
checking that on Linux anyway so doing it for Mac is redundant.

Depends on D19366

For some reason running these via cross-compiled wrench in Mozilla
automation produces a few pixels difference.

Depends on D19367

One job (which runs on a Linux docker worker) cross compiles wrench in
two configurations (regular and headless) and publishes artifacts with
the binaries. The other job (which runs on macOS) downloads the
artifacts and uses them to run the WebRender macOS CI release-mode
scripts, which basically consists of running the WebRender reftest
suite.

Depends on D19368

Similar to the previous patch, this adds two jobs. One that
cross-compiles the binaries produced by cargo test and publishes them
as an artifact. The other job downloads the artifact and runs the
resulting binaries.

Depends on D19369

The shebang at the top of fetch-content doesn't work on macOS because
the path to python3 is not /usr/bin. Using /usr/bin/env doesn't work
properly on all platforms either so instead we invoke the script using
the currently running python executable.

Depends on D19365

Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e9f887f489e5
Allow run-task jobs on docker-worker to use internal tooltool downloads. r=ahal
https://hg.mozilla.org/integration/autoland/rev/f6413e576a21
Provide path to python executable when invoking fetch-content script. r=ahal
https://hg.mozilla.org/integration/autoland/rev/8990eef6c31e
Update osmesa-src to includes servo/osmesa-src#33 for cross-compiling support. r=kvark
https://hg.mozilla.org/integration/autoland/rev/ee403c79877e
Update WR mac CI scripts to allow cross-compiling. r=kvark
https://hg.mozilla.org/integration/autoland/rev/f06721b552a8
Fuzz a couple of mac reftests. r=gw
https://hg.mozilla.org/integration/autoland/rev/028c0ed368b8
Add jobs to cross-compile wrench and use the resulting binaries. r=glandium
https://hg.mozilla.org/integration/autoland/rev/79120d13e713
Add jobs to cross-compile cargo test binaries and run them on macOS. r=glandium

Busted because of bug 1519472. Bug 1527798 will fix it.

Depends on: 1527798
Flags: needinfo?(kats)

Actually, this can reland without waiting for bug 1527798. Now that the docker images are rebuilt, the toolchain jobs won't be triggered, as their cache hash hasn't changed.

No longer depends on: 1527798

Thanks. I'll give it a bit to let the pending toolchain jobs finish at least. But - when this merges to m-c will it trigger the sane docker/toolchain rebuild there? Or does m-c share a cache with autoland?

Pushed by dluca@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/66b4b0400dfd
Allow run-task jobs on docker-worker to use internal tooltool downloads. r=ahal
https://hg.mozilla.org/integration/autoland/rev/a7506a49e09b
Provide path to python executable when invoking fetch-content script. r=ahal
https://hg.mozilla.org/integration/autoland/rev/07db219dde08
Update osmesa-src to includes servo/osmesa-src#33 for cross-compiling support. r=kvark
https://hg.mozilla.org/integration/autoland/rev/24b5ceb9cf78
Update WR mac CI scripts to allow cross-compiling. r=kvark
https://hg.mozilla.org/integration/autoland/rev/cf5abe6fc4d4
Fuzz a couple of mac reftests. r=gw
https://hg.mozilla.org/integration/autoland/rev/9c70d8c9a318
Add jobs to cross-compile wrench and use the resulting binaries. r=glandium
https://hg.mozilla.org/integration/autoland/rev/5118be1ae898
Add jobs to cross-compile cargo test binaries and run them on macOS. r=glandium

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #27)

Thanks. I'll give it a bit to let the pending toolchain jobs finish at least. But - when this merges to m-c will it trigger the sane docker/toolchain rebuild there? Or does m-c share a cache with autoland?

It's all shared.

== Change summary for alert #19655 (as of Wed, 27 Feb 2019 07:00:19 GMT) ==

Improvements:

2% Explicit Memory linux64-qr opt stylo 478,537,695.58 -> 469,034,563.74

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=19655

(In reply to Florin Strugariu [:Bebe] from comment #31)

== Change summary for alert #19655 (as of Wed, 27 Feb 2019 07:00:19 GMT) ==

Improvements:

2% Explicit Memory linux64-qr opt stylo 478,537,695.58 -> 469,034,563.74

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=19655

This bug just added a new test job, which should be unrelated to the linux64-qr AWSY tests. I think we may need some retriggers on the graph to figure out the true source of the improvement.

Flags: needinfo?(fstrugariu)

My mistake sorry. Bug 1527742 added the improvement

Flags: needinfo?(fstrugariu)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: