split mochitest-oth into- mochitest-a11y (a), mochitest-chrome (c1, c2, c3)

RESOLVED FIXED

Status

Testing
General
RESOLVED FIXED
2 years ago
8 months ago

People

(Reporter: jmaher, Assigned: jmaher)

Tracking

({leave-open})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 5 obsolete attachments)

(Assignee)

Description

2 years ago
with runbydir enabled for all of mochitest-chrome, I would like to split oth up.  a11y will be fast, but chrome takes a while and a bit longer with run-by-dir.  I propose 3 chunks for mochitest-chrome.
(Assignee)

Comment 1

2 years ago
this will need:
* buildbot-configs
* trychooser syntax
* treeherder changes

I assume this will ride the trains starting in 44 on trunk and carry on.
Depends on: 1211905
I've filed bug 1211905 for the Treeherder parts (it's easier to track changes if they are in our component). Un CCing for now to reduce bugmail - once you know the buildernames drop a message in that bug :-)
(Assignee)

Comment 3

2 years ago
Created attachment 8670349 [details] [diff] [review]
buildbot configs (1.0)

these are buildbot configs, we still need mozharness changes for this to work, let alone other dependencies.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8670349 - Flags: review?
(Assignee)

Comment 4

2 years ago
Created attachment 8670350 [details]
bbconfig differences.txt
Comment on attachment 8670349 [details] [diff] [review]
buildbot configs (1.0)

Review of attachment 8670349 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me
Attachment #8670349 - Flags: review? → review+
(Assignee)

Comment 6

2 years ago
Created attachment 8670390 [details] [diff] [review]
mozharness work for chrome-chunks (1.0)

armen, this is needed prior to landing the buildbot changes, do let me know if I am overlooking something!
Attachment #8670390 - Flags: review?(armenzg)

Updated

2 years ago
Attachment #8670390 - Flags: review?(armenzg) → review+

Comment 7

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a51d8766dfb
(Assignee)

Updated

2 years ago
Keywords: leave-open
https://hg.mozilla.org/mozilla-central/rev/2a51d8766dfb
(Assignee)

Comment 9

2 years ago
landed code to split up m-oth:
https://hg.mozilla.org/build/buildbot-configs/rev/47df7227fa0f

when the next reconfig happens this will be live.
(Assignee)

Comment 10

2 years ago
backed out https://hg.mozilla.org/build/buildbot-configs/rev/de29b5545a08, builder limits:
tst-linux64-spot-544 has 4131 builders; limit is 4084; 101 percent of max

:catlee, do I just need to find other tst-linux64-spot-544 jobs to disable?
Flags: needinfo?(catlee)
(Assignee)

Comment 11

2 years ago
as seen in this log file:
https://travis-ci.org/mozilla/build-buildbot-configs/jobs/85146131

Updated

2 years ago
Depends on: 1215527
(Assignee)

Comment 12

2 years ago
found some builders and landed:
http://hg.mozilla.org/build/buildbot-configs/rev/1e5812a86db3

tests passed successfully- this will get picked up in the next reconfig, just missed the one from today.
Flags: needinfo?(catlee)
These new jobs are all failing with  "Required config file not set! (use --config-file option)" at the start of the desktop_unittest.py step.
Flags: needinfo?(jmaher)
Flags: needinfo?(armenzg)
Duplicate of this bug: 1173218
(Assignee)

Comment 15

2 years ago
this is the command that is run when this was live:
 /tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols ondemand
 

and the error:
 Required config file not set! (use --config-file option)
 

for a dt5 run, we see this type of command executed:
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite mochitest-devtools-chrome-chunked --total-chunks 8 --this-chunk 5 --blob-upload-branch mozilla-inbound --download-symbols ondemand


what appears to be missing is " --cfg unittests/linux_unittest.py ".  Most likely we would need to add stubs similar to this for mochitest-a11y and mochitest-chrome-chunked:
https://dxr.mozilla.org/build_buildbot-configs/source/mozilla-tests/config.py#876

and for osx and windows as well.  

:armenzg, can you confirm that, I would be happy to give this another go.
Flags: needinfo?(jmaher)

Comment 16

2 years ago
Yes, this is correct. I should have caught this in review.

It would be nice if test-master.sh would catch issues like this.
Flags: needinfo?(armenzg)
(Assignee)

Comment 17

2 years ago
Created attachment 8676886 [details] [diff] [review]
buildbot configs (2.0)

same bbconfig differences, just updated to have --cfg for each job type!
Attachment #8670349 - Attachment is obsolete: true
Attachment #8676886 - Flags: review?(armenzg)

Updated

2 years ago
Attachment #8676886 - Flags: review?(armenzg) → review+
(Assignee)

Comment 18

2 years ago
landed this again:
https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd

will see if a reconfig yields success.
In production: https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd

Comment 20

2 years ago
Backed out:
https://hg.mozilla.org/build/buildbot-configs/rev/a25e920f6d1d

I will file a bug that will preven this happening in the future.

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1445613808/mozilla-inbound_ubuntu64_vm-debug_test-mochitest-a11y-bm51-tests1-linux64-build1.txt.gz
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols true
 in dir /builds/slave/test/. (timeout 1800 secs) (maxTime 1800 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--mochitest-suite', 'a11y', '--blob-upload-branch', 'mozilla-inbound', '--download-symbols', 'true']
 environment:
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  DISPLAY=:0
  HOME=/home/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MOZ_HIDE_RESULTS_TABLE=1
  MOZ_NODE_PATH=/usr/bin/node
  MOZ_NO_REMOTE=1
  NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
  NO_FAIL_ON_TEST_ERRORS=1
  PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PROPERTIES_FILE=/builds/slave/test/buildprops.json
  PWD=/builds/slave/test
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  XDG_SESSION_COOKIE=9ca12473fbb1d023794ffd180000023c-1445617269.523098-45509168
  _=/tools/buildbot/bin/python
 using PTY: False
Required config file not set! (use --config-file option)
program finished with exit code 255
(Assignee)

Comment 21

2 years ago
Armen, I am unclear on how this happened since we had this same error the first time and fixed it in the new patch/review.  some how we are missing the --cfg.
Flags: needinfo?(armenzg)

Comment 22

2 years ago
Before I backed out, I could not determine why it would failed (henced backed out).

This time I've noticed that this was only added for the 'linux' block and not 'linux64'.
Flags: needinfo?(armenzg)
(Assignee)

Comment 23

2 years ago
Created attachment 8680242 [details] [diff] [review]
added configs for all platforms (3.0)

ok, this should do the trick!
Attachment #8676886 - Attachment is obsolete: true
Attachment #8680242 - Flags: review?(armenzg)

Comment 24

2 years ago
Comment on attachment 8680242 [details] [diff] [review]
added configs for all platforms (3.0)

Review of attachment 8680242 [details] [diff] [review]:
-----------------------------------------------------------------

Let's hope this time it works :)
Attachment #8680242 - Flags: review?(armenzg) → review+

Comment 25

2 years ago
Created attachment 8680583 [details]
builder changes
(Assignee)

Comment 26

2 years ago
ok, round 3:
https://hg.mozilla.org/build/buildbot-configs/rev/8af4ed62fb0f
(Assignee)

Comment 27

2 years ago
backed out!
https://hg.mozilla.org/build/buildbot-configs/rev/cb2919a9a732

this time we needed builders to open aurora by adding more chunks.
(Assignee)

Comment 28

2 years ago
the actual patch that was supposed to be backed out:
http://hg.mozilla.org/build/buildbot-configs/rev/3171c91cf580
(Assignee)

Comment 29

2 years ago
this previous patch is for all platforms, on linux32 debug we are hitting timeouts more frequently- for taskcluster (linux64 debug) we are already doing this split- for now lets adjust linux32 debug only.
Blocks: 1233953
Flags: needinfo?(jmaher)
(Assignee)

Comment 30

2 years ago
Created attachment 8711690 [details] [diff] [review]
oth -> a11y/c1/c2/c3 for linux32 debug only
Attachment #8680242 - Attachment is obsolete: true
Flags: needinfo?(jmaher)
Attachment #8711690 - Flags: review?(armenzg)
(Assignee)

Comment 31

2 years ago
Created attachment 8711691 [details]
builder differences for linux32 debug only

and the differences file
Attachment #8670350 - Attachment is obsolete: true
Attachment #8680583 - Attachment is obsolete: true

Comment 32

2 years ago
Comment on attachment 8711690 [details] [diff] [review]
oth -> a11y/c1/c2/c3 for linux32 debug only

Could you also please paste the list_builders_differences output for posterity? Thanks!
Attachment #8711690 - Flags: review?(armenzg) → review+
(Assignee)

Comment 33

2 years ago
landed:
https://hg.mozilla.org/build/buildbot-configs/rev/d9e027f12297
http://hg.mozilla.org/build/buildbot-configs/rev/c06c1be8884f and in production.

Comment 35

2 years ago
Do we want to make this change for linux64 as well?
(Assignee)

Comment 36

2 years ago
we do, I couldn't land this last year because I hit the builder limits on linux64.
No longer blocks: 1233953
(Assignee)

Updated

8 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.