1110982 - tracking bug to get --run-by-dir enabled for 'oth' job on trunk

Status: RESOLVED FIXED

(Testing :: General, defect)

Comment 1

[Vaibhav (:vaibhav1994)]

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=054c7cc31858 - Try push in which some tests are disabled that were failing with run-by-dir previously.

Comment 2

[Vaibhav (:vaibhav1994)]

Try push with more tests disabled to tackle leakchecks: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=59952fa0bb96

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

https://tbpl.mozilla.org/?tree=Try&rev=547c73267a01

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

https://tbpl.mozilla.org/?tree=Try&rev=3807db5aea2f

Comment 5

[Vaibhav (:vaibhav1994)]

Attachment: oth-disabled_tests.patch

The list of disabled tests for complete green push for oth jobs on try.

The green push: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=506a7fcead24

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

this try push shows that no new issues have cropped up:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=14a8a8c29665

14 tests were disabled in that one, pushing a few more times to see if the other failures are still there

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: disable 5 tests for mochitest-chrome, enable run-by-dir for m-oth jobs (1.0)

silence on needinfo? requests for a week yields disabling!  The ASAN leak is ok'd to disable.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a53c8b6ab6df

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a53c8b6ab6df

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: turn_off_runbydir.patch

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8562080 [details] [diff] [review]
turn_off_runbydir.patch

Thanks Joel, was worth a shot :(

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/bf8aff9d15d3

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/bf8aff9d15d3

Comment 14

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8664379 [details] [diff] [review]
rbd_chrome.patch

Review of attachment 8664379 [details] [diff] [review]:
-----------------------------------------------------------------

this look great, thanks for the clean bugs and hard work on tracking this down.

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Note this patch as written will conflict with bug 912121, all DevTools files have moved, and same test manifest lines were also modified.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

thanks jryans, :kaustabh, can you 'hg pull -u' and refresh your patch?  I have removed the checkin-needed until we refresh.

Comment 18

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

Comment 19

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

this had the wrong bug number in the comment, but it landed last week:
https://hg.mozilla.org/mozilla-central/rev/81fca7e4e6ef

Comment 21

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=91d8f8827d83

Comment 22

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=91d8f8827d83

Comment 23

[Kaustabh Datta Choudhury]

Attachment: rbd_chrome.patch

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8669342 [details] [diff] [review]
rbd_chrome.patch

Review of attachment 8669342 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great!

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

ok, this will get us running in run-by-dir for the oth job on debug- we will see a hit on runtime, Ideally we could now chunk chrome to reduce our overall runtime.  the problem is we have a11y in the other job as well as chrome, if we want to chunk chrome we need to change oth to just be a11y, then run chrome in X chunks.

There could be other realistic ways to solve this, I am open to all ideas.

Comment 26

[Jonathan Griffin (:jgriffin)]

Yeah I'm thinking we get rid of mochitest-oth and replace it with separate jobs for chrome and a11y.  This makes a lot of downstream stuff easier as well, if we don't have to worry about different suites being run in the same job.  That's an old optimization that probably doesn't make sense any longer.

Comment 27

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.