1302152 - Make forced artifact builds on buildbot trigger test jobs

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(This is a follow-up to Bug 1278698.)

In this try push we can see that when an artifact build is forced in a buildbot job with --artifact try syntax, the tests requested in the try syntax are not scheduled. I think this is because the mozharness config for artifact builds skips the upload-files step and sendchange step. 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea47a93cdc20&group_state=expanded

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

More info: if I trigger a test job manually (via add new jobs) after the forced artifact build job succeeds on buildbot, the test job runs fine, so maybe all that's missing is the sendchange step?

Log shows that the test job downloads the Firefox binary from my try commit, so the artifact build did get uploaded:
10:11:07     INFO - Downloading https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-ea47a93cdc2004a8311be011e6106d67fcd8c8b0/try-linux/firefox-51.0a1.en-US.linux-i686.tar.bz2 to /builds/slave/test/build/installer.tar.bz2

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Indeed, enabling sendchange fixes the lack of test-triggering on buildbot, but breaks the build job on TC. I'll take a look at modifying _do_sendchange to avoid that.

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=28164236
>16:41:55     INFO - Running main action method: sendchange
>16:41:55     INFO - 'enable_talos_sendchange' is false; skipping
>16:41:55    ERROR - could not determine packageUrl property to use against sendchange. Was it set after 'mach build'?

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ada056a50e8f21bff416938bbb6f405d6dfc182
and
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4f1605c307f39dd1338d3b0598abfeb5d23c339

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1302152 - Make forced artifact builds on buildbot trigger test jobs;

Review commit: https://reviewboard.mozilla.org/r/81902/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/81902/

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8795931 [details]
Bug 1302152 - Make forced artifact builds on buildbot trigger test jobs;

https://reviewboard.mozilla.org/r/81902/#review80706

lgtm. I'm not too familiar with artifact builds. It seems like as far as buildbot is concerned, they are 'linux64 opt' builds and tests. I say that because these artifact variants use the same buildbot builders and upload locations as 'linux64 opt' (debug, asan, etc have their own). Which is probably unrelated here and known already.

::: testing/mozharness/mozharness/mozilla/building/buildbase.py:1954
(Diff revision 1)
> +        if (os.environ.get('TASK_ID') or
> +                (self.buildbot_config
> +                 and self.buildbot_config.get('comments') == 'TaskCluster Job')):

hm, is this how we determine if a mh job is from tc now? I wonder if this should be a helper method. Maybe out of scope for this bug though :)

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8795931 [details]
Bug 1302152 - Make forced artifact builds on buildbot trigger test jobs;

https://reviewboard.mozilla.org/r/81902/#review80772

Jordan's review is more than enough here (I share his question).

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Note also Armen just landed bug 1299702, we should incorporate this change to those configs here (or in a follow-up if that seems more expedient).

Comment 8

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8795931 [details]
Bug 1302152 - Make forced artifact builds on buildbot trigger test jobs;

https://reviewboard.mozilla.org/r/81902/#review80706

> hm, is this how we determine if a mh job is from tc now? I wonder if this should be a helper method. Maybe out of scope for this bug though :)

I just made it up, but I asked around and just checking TASK_ID is good enough. Had a long conversation with dustin that can be summed up with this excerpt:
```
16:26 <maja_zf> dustin: do you think it's possible to be *not* running in TC while having the TASK_ID env var?
16:26 <dustin> no
```

Comment 9

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Made changes to remaining artifact mozharness configs (win, mac, etc)

Nothing broke on try (without --artifact): https://treeherder.mozilla.org/#/jobs?repo=try&revision=f605d8254c4806ca5d94b409b042dfd557921444

Test jobs are being triggered correctly (with --artifact):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a6ad69689be625b6578593f3da2e0cdf4d68b8e
and
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5d6c2e3f4d09e6d8aa747886a2f7a1948ca4b31

Comment 10

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8795931 [details]
Bug 1302152 - Make forced artifact builds on buildbot trigger test jobs;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/81902/diff/1-2/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b04101c385d4
Make forced artifact builds on buildbot trigger test jobs; r=jlund

Comment 12

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/b04101c385d4