1211889 - split mochitest-oth into- mochitest-a11y (a), mochitest-chrome (c1, c2, c3)

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

with runbydir enabled for all of mochitest-chrome, I would like to split oth up.  a11y will be fast, but chrome takes a while and a bit longer with run-by-dir.  I propose 3 chunks for mochitest-chrome.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this will need:
* buildbot-configs
* trychooser syntax
* treeherder changes

I assume this will ride the trains starting in 44 on trunk and carry on.

Comment 2

[Ed Morley [:emorley]]

I've filed bug 1211905 for the Treeherder parts (it's easier to track changes if they are in our component). Un CCing for now to reduce bugmail - once you know the buildernames drop a message in that bug :-)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: buildbot configs (1.0)

these are buildbot configs, we still need mozharness changes for this to work, let alone other dependencies.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: bbconfig differences.txt

Comment 5

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8670349 [details] [diff] [review]
buildbot configs (1.0)

Review of attachment 8670349 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: mozharness work for chrome-chunks (1.0)

armen, this is needed prior to landing the buildbot changes, do let me know if I am overlooking something!

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2a51d8766dfb

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2a51d8766dfb

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

landed code to split up m-oth:
https://hg.mozilla.org/build/buildbot-configs/rev/47df7227fa0f

when the next reconfig happens this will be live.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

backed out https://hg.mozilla.org/build/buildbot-configs/rev/de29b5545a08, builder limits:
tst-linux64-spot-544 has 4131 builders; limit is 4084; 101 percent of max

:catlee, do I just need to find other tst-linux64-spot-544 jobs to disable?

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

as seen in this log file:
https://travis-ci.org/mozilla/build-buildbot-configs/jobs/85146131

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

found some builders and landed:
http://hg.mozilla.org/build/buildbot-configs/rev/1e5812a86db3

tests passed successfully- this will get picked up in the next reconfig, just missed the one from today.

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

These new jobs are all failing with  "Required config file not set! (use --config-file option)" at the start of the desktop_unittest.py step.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

this is the command that is run when this was live:
 /tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols ondemand
 

and the error:
 Required config file not set! (use --config-file option)
 

for a dt5 run, we see this type of command executed:
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite mochitest-devtools-chrome-chunked --total-chunks 8 --this-chunk 5 --blob-upload-branch mozilla-inbound --download-symbols ondemand


what appears to be missing is " --cfg unittests/linux_unittest.py ".  Most likely we would need to add stubs similar to this for mochitest-a11y and mochitest-chrome-chunked:
https://dxr.mozilla.org/build_buildbot-configs/source/mozilla-tests/config.py#876

and for osx and windows as well.  

:armenzg, can you confirm that, I would be happy to give this another go.

Comment 16

[Armen [:armenzg]]

Yes, this is correct. I should have caught this in review.

It would be nice if test-master.sh would catch issues like this.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: buildbot configs (2.0)

same bbconfig differences, just updated to have --cfg for each job type!

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

landed this again:
https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd

will see if a reconfig yields success.

Comment 19

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd

Comment 20

[Armen [:armenzg]]

Backed out:
https://hg.mozilla.org/build/buildbot-configs/rev/a25e920f6d1d

I will file a bug that will preven this happening in the future.

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1445613808/mozilla-inbound_ubuntu64_vm-debug_test-mochitest-a11y-bm51-tests1-linux64-build1.txt.gz
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols true
 in dir /builds/slave/test/. (timeout 1800 secs) (maxTime 1800 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--mochitest-suite', 'a11y', '--blob-upload-branch', 'mozilla-inbound', '--download-symbols', 'true']
 environment:
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  DISPLAY=:0
  HOME=/home/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MOZ_HIDE_RESULTS_TABLE=1
  MOZ_NODE_PATH=/usr/bin/node
  MOZ_NO_REMOTE=1
  NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
  NO_FAIL_ON_TEST_ERRORS=1
  PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PROPERTIES_FILE=/builds/slave/test/buildprops.json
  PWD=/builds/slave/test
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  XDG_SESSION_COOKIE=9ca12473fbb1d023794ffd180000023c-1445617269.523098-45509168
  _=/tools/buildbot/bin/python
 using PTY: False
Required config file not set! (use --config-file option)
program finished with exit code 255

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

Armen, I am unclear on how this happened since we had this same error the first time and fixed it in the new patch/review.  some how we are missing the --cfg.

Comment 22

[Armen [:armenzg]]

Before I backed out, I could not determine why it would failed (henced backed out).

This time I've noticed that this was only added for the 'linux' block and not 'linux64'.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: added configs for all platforms (3.0)

ok, this should do the trick!

Comment 24

[Armen [:armenzg]]

Comment on attachment 8680242 [details] [diff] [review]
added configs for all platforms (3.0)

Review of attachment 8680242 [details] [diff] [review]:
-----------------------------------------------------------------

Let's hope this time it works :)

Comment 25

[Armen [:armenzg]]

Attachment: builder changes

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

ok, round 3:
https://hg.mozilla.org/build/buildbot-configs/rev/8af4ed62fb0f

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

backed out!
https://hg.mozilla.org/build/buildbot-configs/rev/cb2919a9a732

this time we needed builders to open aurora by adding more chunks.

Comment 28

[Joel Maher ( :jmaher ) (UTC -8)]

the actual patch that was supposed to be backed out:
http://hg.mozilla.org/build/buildbot-configs/rev/3171c91cf580

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

this previous patch is for all platforms, on linux32 debug we are hitting timeouts more frequently- for taskcluster (linux64 debug) we are already doing this split- for now lets adjust linux32 debug only.

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: oth -> a11y/c1/c2/c3 for linux32 debug only

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: builder differences for linux32 debug only

and the differences file

Comment 32

[Armen [:armenzg]]

Comment on attachment 8711690 [details] [diff] [review]
oth -> a11y/c1/c2/c3 for linux32 debug only

Could you also please paste the list_builders_differences output for posterity? Thanks!

Comment 33

[Joel Maher ( :jmaher ) (UTC -8)]

landed:
https://hg.mozilla.org/build/buildbot-configs/rev/d9e027f12297

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/build/buildbot-configs/rev/c06c1be8884f and in production.

Comment 35

[Armen [:armenzg]]

Do we want to make this change for linux64 as well?

Comment 36

[Joel Maher ( :jmaher ) (UTC -8)]

we do, I couldn't land this last year because I hit the builder limits on linux64.

Comment 37

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.