1494333 - Retriggering nightly test jobs attempts to schedule a new nightly build

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Ryan VanderMeulen [:RyanVM]]

I noticed a failed OSX Nightly Talos test job today. I tried to retrigger it via Treeherder. I would have expected it to simply schedule a new Talos run using the existing Nightly build, but instead a new Nightly build job was scheduled. This effectively means we can't retrigger failed test jobs on Nightlies at the moment.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Links?

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Not sure what specifically you want a link to, so here's a link to the entire push. The OSX g5 job is what I wanted to retrigger.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=32fb340597628c8eb3cea9b507a073a63fb6b81e

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

So action task
  https://tools.taskcluster.net/groups/IadRzZVgRyGbVC-i_vuIEg/tasks/bFJ2lelIReSEhIPlbTT0uQ
consulted decision task
  https://tools.taskcluster.net/groups/IadRzZVgRyGbVC-i_vuIEg/tasks/IadRzZVgRyGbVC-i_vuIEg
looking for "existing tasks" to match build-macosx64-nightly/opt.  It doesn't find one.  Looking at that decision task's label-to-taskId.json:
  https://taskcluster-artifacts.net/IadRzZVgRyGbVC-i_vuIEg/0/public/label-to-taskid.json
shows that indeed, it did not schedule one.  So, the action task did:

[task 2018-09-26T14:39:49.213Z] Creating task with taskId MUKpK4vdR3GGgAXo4VsoFw for build-macosx64-nightly/opt
[task 2018-09-26T14:39:51.139Z] Creating task with taskId f-1Zt1X-SiWtK4APUZQLxA for test-macosx64-nightly/opt-talos-g5-profiling-e10s

I see the first such task built
  https://tools.taskcluster.net/groups/cdGgEUfVTPyyfYKTyZtFEw/tasks/GNHt9oikQMC4CzgUtvHUvA/details
and some digging around shows it was created by the cron task:
  https://taskcluster-artifacts.net/cdGgEUfVTPyyfYKTyZtFEw/0/public/logs/live_backing.log

So it seems that the action task did not find the tasks scheduled by the cron task.  which makes some sense -- there's code to find previous *actions* and consider them, but that may not be finding crons.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=18f3a99aa5adbd9f85d30172a0ad171d611f62ad

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1494333 - index crons just like artifacts

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8db96096643b
index crons just like artifacts r=Callek

Comment 7

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/8db96096643b

Comment 8

[Andreea Pavel [:apavel]]

Backed put for breaking nightly

Backout: https://hg.mozilla.org/mozilla-central/rev/2f85c2b55cd2f1b241f27b7ad8b9dfe42dfc139c

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Andreea, what was the issue with nightlies?  I assume this is the push?
  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed,busted,exception,retry,usercancel,running,pending,runnable&revision=ddcd7cc2f3cdd88f5f399f4c59d530aceda0d722&selectedJob=204749026

I don't see anything wrong there that's related to action tasks.  What led you to back this out?

Comment 10

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #9)
> Andreea, what was the issue with nightlies?  I assume this is the push?
>  
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&resultStatus=testfailed,busted,exception,retry,usercancel,running,
> pending,
> runnable&revision=ddcd7cc2f3cdd88f5f399f4c59d530aceda0d722&selectedJob=204749
> 026
> 
> I don't see anything wrong there that's related to action tasks.  What led
> you to back this out?

https://taskcluster-artifacts.net/IE8VlyHVQHigDqFgOFFeTw/0/public/logs/live_backing.log

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, thanks!

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/623c34a2c9dc
index crons just like artifacts r=Callek

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/623c34a2c9dc

Comment 14

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/dbba5832a8d7