1516597 - Support toolchain tasks on generic-worker on macOS

Status: NEW

(Firefox Build System :: Task Configuration, task, P5)

Description

[Kartikaya Gupta (email:[e-mail])]

Right now toolchain tasks on generic-worker assume they are running on Windows. But it should be possible to make the code compatible with macOS as well. I have WIPs that do this while I was trying to solve bug 1516568. I ran into a problem with the chain of trust that I don't know how to resolve though. With chain-of-trust enabled on the toolchain task (which it should be), taskcluster complains (see log below). With c-o-t disabled, I can't use the resulting artifact in a downstream task.

https://tools.taskcluster.net/groups/cybkbVamRhK0Aqv1-0qW6A/tasks/N94W8e2vR12CjgU1bU-2xg/runs/0/logs/public%2Flogs%2Flive_backing.log

Will attach my WIPs.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Attachment: WIP part 1 - Make generic_worker_hg_commands macos-friendly

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Attachment: WIP part 2 - Allow toolchain jobs on generic-worker on macos

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Attachment: WIP part 3 - Sample toolchain task for macOS

This toolchain task runs to completion successfully, but only if I comment out the chain-of-trust setting at [1]. Failure to do so results in a TC error (see log in comment 0)

[1] https://searchfox.org/mozilla-central/rev/8a135a9c5a96b59269f544fcaee76d8fd5a7026a/taskcluster/taskgraph/transforms/job/toolchain.py#195

Comment 4

[Kartikaya Gupta (email:[e-mail])]

For posterity here is the actual error relevant bit of the log:

[taskcluster 2018-12-28T02:26:36.641Z] === Task Starting ===
[taskcluster 2018-12-28T02:26:37.118Z] Uploading artifact public/logs/certified.log from file generic-worker/certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-01-25T02:26:18.956Z
[taskcluster 2018-12-28T02:26:37.822Z] Uploading artifact public/chainOfTrust.json.asc from file generic-worker/chainOfTrust.json.asc with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-01-25T02:26:18.956Z
[taskcluster:error] Was expecting attempt to read private chain of trust key as task user to fail - however, it did not!

Comment 5

[Mike Hommey [:glandium]]

Why would we need toolchain tasks running on macOS?

Comment 6

[Justin Wood (:Callek)]

On the chain of trust front, it exists to make sure we know that:

* only users who can affect a build have touched this system (L1 v L3) via automation [since try is wild west]
* The build/task came from an in-tree checkin
* The build/task came from the same revision we're building
* Artifacts are not modified in the wild [e.g. target.exe has the same hash as it had at upload time]

Doing this on OSX hardware is going to be harder since Chain of trust uses a gpg key on the worker itself to sign stuff (in theory we can protect it, but its harder to guarantee aiui), but more importantly the OSX machines are test machines and shared between Try and Mozilla-Central, with the underlying assumption that nothing relies on them downstream.

I would be eager to know, like Mike, the answer to c#5

Comment 7

[Kartikaya Gupta (email:[e-mail])]

The goal here is bug 1516568 - that is, running a bunch of WebRender testing on macOS as part of Firefox CI. To keep it simple I tried to just use generic-worker on macOS to run the appropriate `cargo test` commands and to run WebRender's built-in reftest suite. However this runs into errors because some of the dependencies need things like pkg-config and cmake which are not on the macOS workers. So then to resolve that I wanted to build a toolchain task which basically installs brew and uses that to get pkg-config and cmake, and bundles that into an artifact that the test task can use to satisfy the dependency requirements.

However if this is too much trouble or undesirable I can try pursuing other options. For one thing, this toolchain task would really only ever run once so I could just build the tarball locally and put it in tooltool directly. Or I could take a step back and do something like cross-compiling from a Linux machine and only running the final binary on macOS, although I don't know what kind of issues I'll run into with that.

Comment 8

[Tom Prince [:tomprince]]

There is some ongoing work in support macOS PGO that will enable us to have have chain-of-trust on a small pool of OSX workers. However, if the toolchain is just being used in tests, then I'm not sure that chain-of-trust needs to be enabled for the tasks?