1046047 - Try jobs backing up - need to deprioritise ash

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Pete Moore [:pmoore][:pete]]

Archaeopteryx
11:13:39 simone: hi, do you know what's up with that high win32 test job queue on try (1723 jobs)?
Tomcat|sheriffduty
11:13:51 pmoore: ^
pmoore
11:13:59 lol
simone
11:14:21 not really - the pool seems to be working at full capacity, and there are not boxes which have not been taking jobs for a long time
Tomcat|sheriffduty
11:14:23 pmoore: is this due to the disabled slaves from yesterday ?
pmoore
11:15:08 Tomcat|sheriffduty: i disabled 3 slaves - were more disabled after i left?
11:15:15 Tomcat|sheriffduty: i think 3 slaves should not have too much impact
simone
11:15:21 https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=t-w864-ix
11:15:36 6 disabled slaves in total
pmoore
11:16:09 ah yes, i think we disabled builders yesterday?
11:19:10 http://builddata.pub.build.mozilla.org/reports/pending/pending.html - last graph
Archaeopteryx
11:19:21 "Unknown ? Pushlog failed to load"
pmoore
11:19:48 Archaeopteryx: was that when pushing to hg?
Archaeopteryx
11:20:22 when calling the push url returned by the hg push, let me check if this is due to some local network issue
11:20:58 Archaeopteryx left the room (quit: NickServ (GHOST command used by Archae|real)).
11:21:29 Archaeopteryx [[e-mail]] entered the room.
11:22:07 rhelmer left the room (quit: Ping timeout).
Archaeopteryx
11:22:42 that's what i get on https://tbpl.mozilla.org/?tree=Try&rev=01c45e2577b8
11:22:47 https://tbpl.mozilla.org/?tree=Try&showall=1 says "Loading failed: timeout ..."
11:22:49 strange, after opening the url manually, the latter url works
11:23:14 so I suspect it's a local network issue, sorry
nthomas|away
11:23:21 the windows pending is from ash (same priority as try and other project branches). catlee is pushing m-i over there because ash is being used to test new http proxies in AWS
11:24:47 if it's going on for a while it probably makes sense to make ash lower priority, so at least people get their test results on try
pmoore
11:26:06 nthomas|away: maybe you told me before (apologies if so) - do you know why https://secure.pub.build.mozilla.org/buildapi/self-serve/try only lists 635 pending jobs?
11:26:53 nthomas|away is now known as nthomas
nthomas
11:29:12 I don't know, that's weird. Should be over 2000
pmoore
11:29:43 whew - thought i was going made :)
11:29:44 mad
11:29:50 and https://secure.pub.build.mozilla.org/buildapi/self-serve/ash says "no pending"
11:31:10 unless self-serve only lists manually triggered/retriggered jobs - rather than ones caused by a new push? :/
nthomas
11:31:36 should show them all, but it uses a slightly different query on the db
11:32:10 rhelmer [[e-mail]] entered the room.
pmoore
11:42:14 nthomas: is it straightforward to change priority on ash?
11:42:39 btw, i am now totally confused: http://pastebin.mozilla.org/5708512
nthomas
11:42:44 https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/master_common.py#l43
pmoore
11:43:15 here i tried to read in the pending try jobs into a python dict, and it gave me only 44 jobs
nthomas
11:43:28 that means there are 44 revisions with pending, rather than jobs
pmoore
11:43:44 aaahhh of course
11:43:54 right - and each has a bunch of jobs
11:44:07 slaps self
nthomas
11:44:29 mooaaar beverage
pmoore
11:46:59 Archaeopteryx: nthomas: simone: ok, i'm preparing a patch to de-prioritise ash temporarily

Comment 1

[Pete Moore [:pmoore][:pete]]

Attachment: bug1046047_buildbot-configs_v1.patch

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8464593 [details] [diff] [review]
bug1046047_buildbot-configs_v1.patch

Sending this over to catlee, since he knows what he's up to on Ash.

Comment 3

[Pete Moore [:pmoore][:pete]]

Archaeopteryx: given that this should hopefully resolve itself in the next couple of hours, are you happy for us to wait for the pending queue to drop down?

If you have particular tests that are important, you should be able to bump up the priority of them on self-serve (https://secure.pub.build.mozilla.org/buildapi/self-serve/try).

Were you asking about it because you have particular try jobs in line that you need the results for, or was it a general concern for all people using try?

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Pete Moore [:pete][:pmoore] from comment #3)
> Were you asking about it because you have particular try jobs in line that
> you need the results for, or was it a general concern for all people using
> try?
Just a general concern.

Comment 5

[Pete Moore [:pmoore][:pete]]

Not needed now, afaik.

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 8464593 [details] [diff] [review]
bug1046047_buildbot-configs_v1.patch

Review of attachment 8464593 [details] [diff] [review]:
-----------------------------------------------------------------

Not sure if this is still needed now that we're not hammering ash as much