1461980 - [run-task] Add ability to use artifacts from parent tasks

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

When creating new test suites and tasks, I like to try and avoid mozharness/test packaging in favour of mach/running from the srcdir.

The latter approach is difficult to accomplish if the new task depends on any build artifacts. The mochitest/reftest selftest tasks do this, but it's very crude and not easy to copy over to other types of tasks. Basically all taskgraph does is set an environment variable with the url to the artifact, then there is code in the tasks themselves that will download/set it up.

I'd like a generic way for a task to say "I need these artifacts from this  dependency". Then I'd like the run-task script to automatically download (and possibly extract?) them into a known location.

I believe something like this would be a big step towards not needing mozharness anymore.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This might be a nice intermediate step toward gps's aspiration to have the worker do these downloads for you..

Comment 2

[Andrew Halberstadt [:ahal]]

Ooh, that would be even better! But yeah, unless that proposal is imminent it's probably worthwhile having something like this in the meantime. I have some WIP patches going through try atm.

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1461980 - [taskgraph] Add ability to download artifacts from dependencies in run-task script,

This adds a 'use-artifacts' key to the run_task schema. Tasks can specify artifacts to download like this:

run:
    using: run-task
    use_artifacts:
        build:
            - target.tar.bz2
            - target.common.tests.zip
            - target.mochitest.tests.zip

This will cause the run-task script to download those three artifacts from the task's 'build' dependency.
If the task doesn't have a 'build' dependency, taskgraph generation will error. The artifacts will be
downloaded into $USE_ARTIFACT_PATH. It is up to the task to do whatever extracting/setup may be required.
E.g this setup could go in the task's command.

At this time, only 'run-task' tasks using docker-worker are supported.

Review commit: https://reviewboard.mozilla.org/r/244670/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/244670/

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1461980 - [taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests,

Instead of downloading the build artifacts (rather hackily) in moztest.fixtures, this now happens
directly in the taskgraph module via the run-task script.

For now extraction and setup happens in the task's command key. It might be a good idea to figure
out a syntax to tell run-task to do this extraction, e.g something like:

run:
    using-artifacts:
        build:
            target.tar.bz2:
                extract: true
                path: /home/worker/build
                name: firefox

But for now I wanted to avoid this extra complexity, so maybe it could be done in a follow-up.

Review commit: https://reviewboard.mozilla.org/r/244672/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/244672/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8977041 [details]
Bug 1461980 - [taskgraph] Add ability to download artifacts from dependencies in run-task script,

https://reviewboard.mozilla.org/r/244670/#review251516

::: taskcluster/taskgraph/transforms/job/run_task.py:34
(Diff revision 1)
>      # if true, perform a checkout of a comm-central based branch inside the
>      # gecko checkout
>      Required('comm-checkout'): bool,
>  
> +    # maps a dependency to a list of artifact names to use from that dependency.
> +    # E.g: {"build": ["target.tar.bz2"]}

It might be worth mentioning here where these will be downloaded.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8977042 [details]
Bug 1461980 - [taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests,

https://reviewboard.mozilla.org/r/244672/#review251524

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8977041 [details]
Bug 1461980 - [taskgraph] Add ability to download artifacts from dependencies in run-task script,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/244670/diff/1-2/

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8977042 [details]
Bug 1461980 - [taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/244672/diff/1-2/

Comment 9

[Andrew Halberstadt [:ahal]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=668a825b8bbdf08e16607544e9f87da024b63663

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aceeafcc22c9
[taskgraph] Add ability to download artifacts from dependencies in run-task script, r=dustin
https://hg.mozilla.org/integration/autoland/rev/bbeba6126822
[taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests, r=dustin

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/aceeafcc22c9
https://hg.mozilla.org/mozilla-central/rev/bbeba6126822

Comment 12

[Gregory Szorc [:gps]]

Hah - I had kind of already written these patches. Although I never submitted them for review. That's on me.

The main difference from what I wrote is that I had support for validating the SHA-256 of artifacts. And I think I had support for extracting archives. We should be able to incorporate that into this scheme as follow-ups easily enough though.

Thanks for implementing this. It's a *really* useful feature. It will likely be replaced by features in the workers themselves. But until that day arrives, this bridges the gap quite nicely.