1473541 - Use the fetch-content script to download grcov for running test suites

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Gabriel Vîjială [:gabriel-v]]

Use the taskcluster/scripts/misc/fetch-content script to download the grcov artifact, instead of using "self.download_file" and "tarfile.extractall" in testing/mozharness/mozharness/mozilla/testing/codecoverage.py.

That script has to be pulled from python/mozbuild/mozbuild/action/test_archive.py

Comment 1

[Gabriel Vîjială [:gabriel-v]]

For Windows, python3 should be at c:\mozilla-build\python3\python.exe.

Comment 2

[Andrew Halberstadt [:ahal]]

I'm making a few changes to 'fetches' in bug 1468812, though the 'fetch-content' script won't change much so it should be tangential. Figured it was worth mentioning though.

Comment 3

[Gabriel Vîjială [:gabriel-v]]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
> I'm making a few changes to 'fetches' in bug 1468812, though the
> 'fetch-content' script won't change much so it should be tangential. Figured
> it was worth mentioning though.

Thanks for mentioning it. Could you look at these patches to see if your changes are incompatible with the hacks I made to use "fetches" for "test" tasks? https://reviewboard.mozilla.org/r/255000/diff/1#index_header

Comment 4

[Andrew Halberstadt [:ahal]]

There will be a few minor conflicts, but shouldn't be hard to resolve. I actually wrote a similar patch (that I was planning to finish up after the one in bug 1468812 lands):
https://hg.mozilla.org/try/rev/d77de0cbb522668bfa8894dabd12313e3ecd0c30

I do want to start using 'fetch-content' in all test/mozharness based tasks though. So if you are inclined, it would be nice if that could live in a more generic place than codecoverage.py. But again, this is something I could change later.

If this is low priority, it might be easier to wait for me to solve this generically. But if you need it landed soonish or the patches are ready to go then feel free to land and I can rebase on top.

Comment 5

[Marco Castelluccio [:marco]]

This is partly done with https://hg.mozilla.org/integration/autoland/rev/bc10207e894d. Linux is going through the first branch (https://hg.mozilla.org/integration/autoland/rev/bc10207e894d#l2.22) which is what we want, Windows isn't yet (and we haven't checked Mac).

Comment 6

[Intermittent Failures Robot]

2 failures in 760 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* android-4-0-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1473541&startday=2018-07-23&endday=2018-07-29&tree=trunk

Comment 7

[Gabriel Vîjială [:gabriel-v]]

This will be fixed by :ahal through this patch [1] for this bug [2].

[1]: https://phabricator.services.mozilla.com/D3651
[2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1482344

Comment 8

[Andrew Halberstadt [:ahal]]

Note: I haven't verified that fetch-content is being used for all ccov tasks in that patch. I know there is at least one worker-type (releng-hardware) where fetch-content doesn't yet exist. There may be others. In that case the manual downloading will still be used and you'll see the string "fetch-content script not found, downloading manually".

That patch does assume python 3 is available however (which I believe all the dependent bugs got fixed). I don't see any py3 failures in my try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa78b8866b39da7aaf9fc3d45df8195dd66975c4&filter-tier=1&filter-tier=2&filter-tier=3

Comment 9

[Gabriel Vîjială [:gabriel-v]]

(In reply to Andrew Halberstadt [:ahal] from comment #8)
> Note: I haven't verified that fetch-content is being used for all ccov tasks
> in that patch. I know there is at least one worker-type (releng-hardware)
> where fetch-content doesn't yet exist. There may be others. In that case the
> manual downloading will still be used and you'll see the string
> "fetch-content script not found, downloading manually".

There's one more coverage platform: macos. Since it's disabled, fixing it will probably be a follow up part of bug 1399394.

I confirm that your patch works in my own try push.

Passing this bug over to you. Thanks!

Comment 10

[Marco Castelluccio [:marco]]

It looks like Mac is broken: https://treeherder.mozilla.org/#/jobs?repo=try&revision=62c4c58af9cfaebd1d8820d7be3f97093436903f&filter-searchStr=mac.

Comment 11

[Marco Castelluccio [:marco]]

I've filed bug 1490247 for Mac.

Comment 12

[Marco Castelluccio [:marco]]

And I guess we can mark this as FIXED now.