1488313 - Nightly valgrind testing rebuilds and clobbers index for the linux64 build job.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Justin Wood (:Callek)]

In doing unrelated work [1] I found that we rebuild linux64 opt every time we do valgrind testing... 

e.g. https://tools.taskcluster.net/groups/Dcy_LEynSemmeIpMODLhOA was found for push id 34555 which corresponds to https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8284cddccf78566ca4dc45272940ccae4b1150df

The target tasks only show the mochitest valgrinds...

[1] - https://github.com/mozilla-releng/measuring_ci/issues/33

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

jmaher, can you clarify our intentions for mochitest-valgrind ?

There's a valgrind-linux64-valgrind/opt job on each m-c push [1], which runs ./mach valgrind-test via a mozharness action, and sets taskcluster routes which don't interfere with the main Linux64 opt (build-linux64/opt) [2]. 

The periodic Vg cron creates a second build-linux64/opt [3], which the mochitest-valgrind tests download and use. The bug here is that the build overwrites some routes of the on-push build, including [4].  

So I'm wondering if the mochitest-valgrind tests are using the right build or no. If they are it seems like there's some task graph generation issue where the previous opt build isn't used.

aside: anyone looking on treeherder should enable tier3 to see the mochitest-valgrind tests [5]

[1] https://tools.taskcluster.net/groups/G8qhxHILS26mCvgdY_8tFg/tasks/M9f_LAe5QX-v19Bhg35hSQ/details
[2] https://tools.taskcluster.net/groups/G8qhxHILS26mCvgdY_8tFg/tasks/Cix9r4t2Rwea4uibDWFnjw/details
[3] https://tools.taskcluster.net/groups/Dcy_LEynSemmeIpMODLhOA/tasks/aWY8Py8gS2WYU2eSgdJGGw/details
[4] index.gecko.v2.mozilla-central.revision.8284cddccf78566ca4dc45272940ccae4b1150df.firefox.linux64-opt
[5] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8284cddccf78566ca4dc45272940ccae4b1150df&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=valgrind&selectedJob=197060448

Comment 2

[Tom Prince [:tomprince]]

If we do want to use the `build-linux64/opt for the mochitest-valgrind jobs, then we probably want to do something like rel-pro, where the valgrind-cron tasks pulls in the build-linux64-opt build from the on-push graph.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

interesting observations.  Valgrind tests should be using a valgrind build.  Our goal is to keep these running regularly, but not per commit.  This is where the cronjob comes in for the build which should then kick off all the tests.

It is much nicer to schedule per-commit on mozilla-central (<10/day) instead of a cronjob.  I don't think there is enough support for doing this which means running only the crontab and the related valgrind test jobs once/day.

Comment 4

[Tom Prince [:tomprince]]

Attachment: Bug 1488313: [taskgraph] Move some functions for interacting with existing taskgraph to a more generic location; r?aki

Comment 5

[Tom Prince [:tomprince]]

Attachment: Bug 1488313: [taskgraph] Add options to reuse on-push tasks in cron graphs; r?aki,dustin

Comment 6

[Tom Prince [:tomprince]]

Attachment: Bug 1488313: [taskgraph] Reuse on-push build in nightly valgrind graphs; r?dustin

Comment 7

[Tom Prince [:tomprince]]

Attachment: Bug 1488313: [taskgraph] Only look for try parameters in on-push decision tasks; r?dustin

This avoids the surprising result of try_task_config.json overriding explicit
--target-tasks-method from a cron task run against a try push.

Comment 8

[Tom Prince [:tomprince]]

Attachment: Bug 1488313: [taskgraph] Don't notify sherrifs on cron decision task failures on try; r?dustin

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a9500cb7d276
[taskgraph] Move some functions for interacting with existing taskgraph to a more generic location; r=aki
https://hg.mozilla.org/integration/autoland/rev/b14a6538211f
[taskgraph] Add options to reuse on-push tasks in cron graphs; r=dustin,aki
https://hg.mozilla.org/integration/autoland/rev/e7778d636a11
[taskgraph] Reuse on-push build in nightly valgrind graphs; r=dustin

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/993025739f8d
[taskgraph] Only look for try parameters in on-push decision tasks; r=dustin
https://hg.mozilla.org/integration/autoland/rev/dcfe21ab348e
[taskgraph] Don't notify sherrifs on cron decision task failures on try; r=dustin

Comment 11

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/a9500cb7d276
https://hg.mozilla.org/mozilla-central/rev/b14a6538211f
https://hg.mozilla.org/mozilla-central/rev/e7778d636a11
https://hg.mozilla.org/mozilla-central/rev/993025739f8d
https://hg.mozilla.org/mozilla-central/rev/dcfe21ab348e