1286839 - pulse_actions - Regressed backfilling jobs

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

It seems that ashiue tried to backfill a talos pgo job [1], however, the output from the 'Sch' job [2] does not clearly indicate if it worked or not.

Perhaps it is related that we don't have PGO builds on every push, however, the output doesn't say either way.

ashiue my apologies for any inconveniences this might have caused you.
Please let me know if there's any other information you would like to add to help me understand and how to better fix it.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Ubuntu%20HW%2012.04%20x64%20mozilla-inbound%20pgo%20talos%20tp5o&tochange=752282c3e152f3c56de2c66dd56cb05e79b05807
[2] https://tc-gp-public-31d.s3-us-west-2.amazonaws.com/ateam/pulse-action-dev/906e0a40-66cd-463e-9497-174bae3b2f72

Comment 1

[Alison Shiue]

Hi Armen,

Thank you for filing this bug!
Because it always shows "Request sent to backfill jobs", and the green message will mislead me to think the retrigger works. 
After I wait for a moment and refresh the page but unable to find the missing test, I just retrigger again and again...

It would be very useful if the page can show user the real action result.


Does opt build also have this kind of issue?
I am trying to backfill a cart summary opt e10s test to 4191df8e4b8e[1], but it does not work.

[1] https://treeherder.mozilla.org/#/jobs?repo=fx-team&group_state=expanded&tochange=99b859906a9ca1e86c31415f98c1f8094c4f67d5&fromchange=7d431089a2d2c584bea146b354b06026c80291a1&filter-searchStr=OS%20X%2010.10%20opt&selectedJob=10260426

Comment 2

[Armen [:armenzg]]

Thanks for pointing it out.
I will hopefully look into it today or Monday.

Comment 3

[Armen [:armenzg]]

I fixed this [1]. It seems I introduced a regression on Wednesday [2]
You can see on the latest 'Sch' that the output shows more information [3][4]
You can now see on the output [5] that we're scheduling Linux pgo builds [6]

The bug was affecting everything not just talos pgo job backfilling.

I've also noticed that this code is suboptimal specially for PGO jobs since we trigger the build and *hope* for the talos job to be executed. I will fix this in a follow up bug (bug 1287151).

I've also filed bug 1287149 to improve determining what actually gets scheduled.

[1] https://github.com/mozilla/mozilla_ci_tools/commit/2ef308ffcdf25e3060d8a881d26a7168e2ac6b2e
[2] https://github.com/mozilla/mozilla_ci_tools/commit/6726baea42f9a6c7a9eefb93afaaaab5113c6c7f
[3] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=752282c3e152f3c56de2c66dd56cb05e79b05807&filter-searchStr=Sch&selectedJob=31851815
[4] https://tc-gp-public-31d.s3-us-west-2.amazonaws.com/ateam/pulse-action-dev/f7e481cf-1088-4257-aa60-e3b970949cef
[5]
mozci We will trigger 1) 'Linux x86-64 mozilla-inbound pgo-build'
mozci instead of 2) 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o'
mozci We need to trigger the build job once (1) in order to be able to run the test job (2).
mozci After (1) is done and if no coalesccing happens the test jobs associated with it will be triggered.
[6] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Linux%20x86-64%20mozilla-inbound%20pgo-build&tochange=752282c3e152f3c56de2c66dd56cb05e79b05807&selectedJob=31788678

Comment 4

[Armen [:armenzg]]

I've inspected the logs and you were the only person affected.
The bad deploy was at 6:41 on Wednesday [1]

[1] Jul 13 06:41:19 pulse-actions heroku/api: Deploy 08ddfce by [e-mail]

Jul 13 19:12:08 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Windows 7 32-bit autoland talos chromez-e10s' 
Jul 13 19:12:49 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Windows 7 32-bit autoland talos chromez-e10s' 
Jul 13 19:30:31 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 autoland pgo talos tp5o' 
Jul 13 20:21:58 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 20:24:25 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 20:26:31 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 20:27:06 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 20:32:02 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 autoland pgo talos tp5o' 
Jul 13 22:02:21 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o' 
Jul 13 22:02:32 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o' 
Jul 13 22:02:44 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o' 
Jul 13 22:02:56 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o' 
Jul 13 22:03:06 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Ubuntu HW 12.04 x64 mozilla-inbound pgo talos tp5o' 
Jul 13 22:19:23 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:19:34 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:19:46 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:19:56 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:20:06 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:20:17 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:26:57 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:50:52 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:58:11 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:58:21 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:58:32 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:58:42 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:58:52 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 22:59:03 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 13 23:25:53 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound talos svgr-e10s' 
Jul 13 23:26:07 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound talos svgr-e10s' 
Jul 13 23:27:17 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound talos svgr-e10s' 
Jul 13 23:59:25 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound talos svgr-e10s' 
Jul 14 01:02:08 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 14 01:23:46 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 14 20:40:16 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 14 20:41:57 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s' 
Jul 14 20:50:11 pulse-actions app/worker2.1: treeherder_job_action	 INFO:	 Backfill action requested by [e-mail] for 'Rev7 MacOSX Yosemite 10.10.5 fx-team talos svgr-e10s'