984069 - Verify that trychooser's tryload is looking at the right names for pending jobs and adding them up into the correct buckets

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

This will require either being able to see the source that generates http://builddata.pub.build.mozilla.org/reports/pending/pending_compile_try.txt and http://builddata.pub.build.mozilla.org/reports/pending/pending_test_try.txt and http://builddata.pub.build.mozilla.org/reports/pending/pending_test_nontry.txt, or perhaps (if, as I suspect, you would need to see both the source and a database dump) will require looking at those three URLs during the middle of a weekday when everything is pending long enough to catch it in there.

One (minor one) I did see tonight is that winxp/win7 test load should be "load_try_test["winxp-ix"] + load_nontry_test["winxp-ix"] + load_nontry_test["winxp"]" because despite using the same winxp-ix slaves, tests on the addon-sdk tree are listed as winxp/win7 rather than winxp-ix/win7-ix.

win64 is probably wrong: I'm pretty sure there's one pool of win64 slaves, and both win32 and win64 should be the sum of win32+win64, and since we stopped doing win64 tests it should be N/A.

Depending on how you look at it, linux/emulator tests should include fedora* (debug b-c and emulator reftests) and ubuntu*-hw (talos).

Comment 1

[Phil Ringnalda (:philor)]

So far, the only thing else I've seen from loading them at random times is that something is getting miscounted as try builds on linux-hp, since it will claim 85 pending at times when there are only a dozen. Maybe fedora* and ubuntu*-hw? I've never seen them mentioned as pending tests.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The problem is that I have an old, hardcoded, mapping of builder names to platforms. It quickly gets out of date.

Much better would be to redo it using
  http://builddata.pub.build.mozilla.org/reports/allthethings.json.gz  (also not .gz but 10MB!)
which catlee set that up recently. It checks for changes in buildbotcustom, buildbot-config, and tools repos every 15 minutes and dumps information out of actual master instances, so it's stays up to date. We can then reliably map from buildername to a slavepool, which is similar to a platform. With some human friendly naming for pools, the pending/running can get put into the text files trychooser consumes, and also be plotted using flot/d3/some client-side plotting (so that it scales automatically as pools come and go).

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Update trychooser for new platform names

I've fixed up my pending list -> slave pool mapping, and simplified it quite a bit. eg all the flavours of linux build on 'linux64'; similarly we only have win64 compile slaves. There are also includes a few consistency changes (s/mac10.6-rev4/mac10.6/), s/win8/win8-ix/), and drops tegra (a little early, but we're nearly there). Adds support for linux test running on both hw and vm.

This would get deployed at around the same time as I update http://builddata.pub.build.mozilla.org/reports/pending/ with the matching changes.

Comment 4

[Phil Ringnalda (:philor)]

Comment on attachment 8447797 [details] [diff] [review]
[tools] Update trychooser for new platform names

Sometime in the future, I look forward to someone coming up with a Grand Unified Theory Of When To Sum And When To Max (I vaguely think summing linux tests is probably wrong, but don't much care). Meantime, this'll do fine.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8447797 [details] [diff] [review]
[tools] Update trychooser for new platform names

https://hg.mozilla.org/build/tools/rev/f55676fd01cf

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

http://builddata.pub.build.mozilla.org/reports/pending/pending.html & running.html, so we have accurate conversion to slave pool. Trychooser is updated so that we use that.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

This was already deployed. In other news, backend which assigns jobs to pools now follows allthethings.json, so it stays up to date without intervention (new platforms would need consideration in try chooser).