692421 - enable rev4 10.6 jobs in buildapi again

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Ed Morley [:emorley]]

According to http://build.mozilla.org/builds/pending/pending.html

Pending non-try test jobs:
{
mac10.6-rev4 (1480)
  952 mozilla-inbound
  143 mozilla-central
  120 ionmonkey
   87 mozilla-aurora
   54 services-central
   54 jaegermonkey
   50 release-mozilla-beta-macosx64-talos
   10 fx-team
   10 Unknown
}

Pending try test jobs:
{
mac10.6-rev4 (531)
  531 try
}

I realise there are more rev4 machines on the way (bug 690236 and dependants), but the above backlog is just from the last 24hours - over the next day or two it's going to get even worse.

I propose temporarily (until the new machines are in production) disabling the rev4 runs from all trees other than mozilla-central - which judging by the numbers above, will reduce the workload by ~90%. Plus then deleting all pending jobs from non-mozilla-central trees.

Comment 1

[Armen [:armenzg]]

The rev4 slaves are all locked to jhford's test master.
Let's wait for him to come online and let us know what the best approach is.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

They are on so we can watch for results and variance of results but they can (and should) be hidden from the default tbpl view. I'm looking into doing this.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

They can't be hidden from the tbpl view until at least one build (per builder) has run.

Comment 4

[Phil Ringnalda (:philor)]

Hiding won't solve the worse problem, that they are making it impossible to reason about the state of the infra load and the possibility of missing slaves from the numbers in tbpl's Infrastructure menu. Running one round of jobs will temporarily solve that, too, since the 2000 will coalesce into 20 per tree, but if we're going to do this for a few weeks and they'll be constantly backlogged and constantly coalescing lots of scheduled jobs, excluding them from whatever db drives builds-pending.js (or from builds-pending.js itself if its a db that does other things with them) would probably be worthwhile.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

I have attached all 10 machines to production masters, so they should be working through the jobs as soon as they can.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

FYI, we can easily hide matches to '%Rev4%' in the pending tables and graphs if that is appropriate.

Comment 7

[Armen [:armenzg]]

catlee has fixed this by filtering the Rev4 jobs:
http://hg.mozilla.org/build/buildapi/rev/648c1453007c

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

We should revert these changes now that we have a ton of 10.6 rev4 machines imaged up

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Backed out in http://hg.mozilla.org/build/buildapi/rev/93151b99bcfb

There's a backlog on try still.

Comment 10

[Armen [:armenzg]]

It's looking good now:
http://hg.mozilla.org/build/buildapi/rev/f2423342e58d

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #10)
> It's looking good now:
> http://hg.mozilla.org/build/buildapi/rev/f2423342e58d

That revision seems to add back in the rev4 minis, but still excludes them from the query. Is that intentional? How far away are we from resolving this again?

Comment 12

[Armen [:armenzg]]

Attachment: add rev4 back

Comment 13

[Armen [:armenzg]]

Comment on attachment 566925 [details] [diff] [review]
add rev4 back

coop, yesterday I made a mess of backouts of backouts.

Right now, rev4 machines are showing up in pending jobs and wait times email.

If we want to avoid that the right patch to backout is this one:
http://hg.mozilla.org/build/buildapi/rev/ce8590163c64

I hope it now makes a little bit more of sense.

Notice that we have a lot of hung rev4 machines in bug 694431 yet to be determined.
We probably would want to disable rev4 reporting if bug 694431 does not get resolved soon.

Comment 14

[Armen [:armenzg]]

The core issue in bug 694431 got resolved. It will need to be puppetized though.

Pending jobs looks good.

Comment 15

[Phil Ringnalda (:philor)]

Not sure entirely how much stock I'd put in pending numbers, since another bug 694251 instance took 110 jobs out of the list this morning.