1274507 - pulse_actions - 'Add new Jobs' didn't work for Windows XP pgo + tests on Windows 7 pgo

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

I had to add Windows XP pgo as build to https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=c403ac05b8f42e44a5f7a0c98c2bf190b28706b1 and used the 'Add new Jobs' feature, selected that build and the mochitests bc1-bc7+oth on Windows 7 pgo, but nothing showed up as pending in 30 minutes after the request. asalega in #buildduty was able to request and start them.

Comment 1

[Ed Morley [:emorley]]

Armen, I'm presuming Sebastian just needs his email whitelisting for 'trigger runnable jobs' in pulse_actions? (trigger_runnable_jobs doesn't require the Treeherder is_staff bit, so this isn't caused by bug 1208855).

https://github.com/mozilla/pulse_actions/blob/61ed7be40fc56349c2b0e9ae7ec6112ebe6df4f7/pulse_actions/utils/misc.py#L16

The UX here isn't great - I guess the best solution is just to wait for bug 1273092 so Treeherder knows whether the request will succeed in advance and show appropriate error messages either way.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

I have successfully requested new jobs in the past. Now there are two pgo builds running for Windows XP and one for Windows 8, but I can't see whom requested it (might be possible KWierso was also involved).

Comment 3

[Armen [:armenzg]]

I have removed whitelisting to prevent this confusion.
bug 1245943 will help with showing what pulse_actions is doing to developers.

I see that mozci does not recognize them as valid builders [3]
I don't see those builders on [1]
However I can see them on [2] (which is the real data source).

This probably means that there is a new bug introduced in mozci.
I will look into it.

[1] http://people.mozilla.org/~armenzg/permanent/all_builders.txt
[2] https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json
[3]
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 New jobs requested by [e-mail] for https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=c403ac05b8f42e44a5f7a0c98c2bf190b28706b1 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 List of builders: 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-1 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-2 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-3 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-4 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-5 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-6 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-7 
May 19 23:58:46 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows XP 32-bit mozilla-inbound pgo test mochitest-other 
May 19 23:58:46 pulse-actions app/worker2.1: mozci	 WARNING:	 Buildername Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-1 is *NOT* valid. 
May 19 23:58:46 pulse-actions app/worker2.1: mozci	 INFO:	 Check the file we just created builders.txt for a list of valid builders. 
May 19 23:58:47 pulse-actions app/worker2.1: mozci	 WARNING:	 Buildername Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-5 is *NOT* valid. 
May 19 23:58:47 pulse-actions app/worker2.1: mozci	 INFO:	 Check the file we just created builders.txt for a list of valid builders. 
May 19 23:58:47 pulse-actions app/worker2.1: mozci	 WARNING:	 Buildername Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-3 is *NOT* valid. 
May 19 23:58:47 pulse-actions app/worker2.1: mozci	 INFO:	 Check the file we just created builders.txt for a list of valid builders. 
May 19 23:58:47 pulse-actions app/worker2.1: mozci	 WARNING:	 Buildername Windows XP 32-bit mozilla-inbound pgo test mochitest-browser-chrome-7 is *NOT* valid. 
May 19 23:58:48 pulse-actions app/worker2.1: mozci	 INFO:	 Check the file we just created builders.txt for a list of valid builders.

Comment 4

[Ed Morley [:emorley]]

Ah thank you :-)

Comment 5

[Armen [:armenzg]]

aryx, thank you very much for reporting. It could easily have taken months before anyone would have noticed.

I've fixed the root issue, fixed the alerting, refactored the code to make it easier to make changes to and added a test to catch it:
https://github.com/mozilla/mozilla_ci_tools/commit/9f14aebcf3255a12114076098ef6ce8b0ed5ea8d

This was a rather edge case since not a lot of people request jobs for pgo, thus, making it hard to notice how long it's been broken.

I will make a deployment on Tuesday when I come back to work.

Comment 6

[Ed Morley [:emorley]]

This is what I can quick service!

Thank you Armen :-)

Comment 7

[Armen [:armenzg]]

lol thanks for taking the time to say thanks!
The code had never been written properly since Buildbot's relationships are very full of exceptions.
I just wanted to cry with this patch. Not the best patch to work on a Friday!

Anyways! Long weekend is here! Time to rest. See you Tuesday

PS = I will deploy on Tuesday since I don't want anything breaking while I'm away.

Comment 8

[Ed Morley [:emorley]]

Enjoy your weekend :-)

Comment 9

[Armen [:armenzg]]

I've deployed the change.
It seems that it is working from manual testing [1][2][3]

[1] https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/build/111959449
[2] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=8e697c3542ffbe810d6ecc9912a510216471e99b&filter-searchStr=windows%207%20pgo%20browser-chrome-1&group_state=expanded
[3]
May 24 08:36:07 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 New jobs requested by [e-mail] for https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=8e697c3542ffbe810d6ecc9912a510216471e99b 
May 24 08:36:07 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 List of builders: 
May 24 08:36:07 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 7 32-bit mozilla-inbound pgo test mochitest-browser-chrome-1 
May 24 08:38:28 pulse-actions app/worker2.1: mozci	 INFO:	 We have the necessary files to trigger the downstream job. 
May 24 08:38:28 pulse-actions app/worker2.1: mozci	 INFO:	 We have the necessary files to trigger the downstream job. 
May 24 08:38:28 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 We don't have anything to schedule through TaskCluster 
May 24 08:38:28 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 We're going to schedule these builders via Buildapi. 
May 24 08:38:28 pulse-actions app/worker2.1: mozci	 INFO:	 ==> We want to trigger 'Windows 7 32-bit mozilla-inbound pgo test mochitest-browser-chrome-1' a total of 1 time(s). 
May 24 08:38:28 pulse-actions app/worker2.1: mozci	 INFO:	  
May 24 08:38:28 pulse-actions app/worker2.1: root	 INFO:	 Message {u'resultset_id': u'32184', u'buildernames': [u'Windows 7 32-bit mozilla-inbound pgo test mochitest-browser-chrome-1'], u'project': u'mozilla-inbound', u'requester': u'[e-mail]', u'version': 1}, took 141.54582119 seconds to execute