969988 - ASan is the only Linux platform running the separate jittest suite on try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

I have a vague memory of a bug where we thought that we were turning off having jittest run on both the ubuntu and the fedora slaves, which I suspect is the reason why https://tbpl.mozilla.org/?tree=Try&rev=bc445a9dbee6&showall=1&jobname=jit got no Linux jittests other than on ASan.

Comment 1

[Chris AtLee [:catlee]]

So we want jittests to run on all linux platforms on try, not just ASan

Comment 2

[Dan Minor [:dminor]]

It looks like the problem is at [1] which I guess was added to prevent scheduling on both fedora and ubuntu slaves on Cedar. This condition should probably be moved to where we add the tests for Cedar so that it doesn't affect try.

[1] http://hg.mozilla.org/build/buildbot-configs/file/8cd7c81484d5/mozilla-tests/config.py#l1774

Comment 3

[Dan Minor [:dminor]]

Attachment: Make logic to skip Fedora slaves only apply to Cedar

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 8383685 [details] [diff] [review]
Make logic to skip Fedora slaves only apply to Cedar

Review of attachment 8383685 [details] [diff] [review]:
-----------------------------------------------------------------

our configs suck

Comment 5

[Dan Minor [:dminor]]

Thanks! pushed to https://hg.mozilla.org/build/buildbot-configs/rev/219b277dba5a

Comment 6

[Jordan Lund (:jlund)]

buildbot-configs patch is merged and live in production! :)

Comment 7

[Dan Minor [:dminor]]

My patch did not work, these are still running on ASAN only.

Comment 8

[Dan Minor [:dminor]]

I'm happy to redo this patch, but I could use some help identifying the root cause of why these are only being scheduled on ASAN for try. Scheduling for cedar is fine.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Dan Minor [:dminor] from comment #8)
> I'm happy to redo this patch, but I could use some help identifying the root
> cause of why these are only being scheduled on ASAN for try. Scheduling for
> cedar is fine.

Looks like the first check of the block is wrong:
https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla-tests/config.py#L1826

>>> 1 in ([2,3] or [4,5])
False
>>> 1 in ([2,3] or [4,5,1])
False

You should probably check those separately, eg:
if platform not in cedar_platforms or platform not in try_platforms:
  continue

Comment 10

[Dan Minor [:dminor]]

D'oh, thanks for catching that. And here I thought it was something mysterious with buildbot.

Comment 11

[Dan Minor [:dminor]]

Attachment: Fix initial platform check

Switch not in (A or B) to not in A and not in B. If A is not an empty list, (A or B) returns A, so the original code never checked inclusion in B.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8392842 [details] [diff] [review]
Fix initial platform check

Review of attachment 8392842 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1822,5 @@
>                          pass
>  
>  # Enable jittests on cedar https://bugzilla.mozilla.org/show_bug.cgi?id=912997
>  for platform in PLATFORMS.keys():
> +    if platform not in BRANCHES['cedar']['platforms'] and platform not in BRANCHES['try']['platforms']:

I think you want "or" not "and". Otherwise you can't control them for cedar and try independently. r=me if you fix that on landing.

Comment 13

[Dan Minor [:dminor]]

Attachment: Fix platform check

I split the code to handle cedar and try separately. I think this will fix the scheduling problem and be easier to read.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 8395635 [details] [diff] [review]
Fix platform check

Review of attachment 8395635 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me now

Comment 15

[Dan Minor [:dminor]]

Thanks, pushed to https://hg.mozilla.org/build/buildbot-configs/rev/8f2ea5f07b81

Comment 16

[[e-mail] (:bhearsum)]

in production

Comment 17

[Dan Minor [:dminor]]

Looks like my new patch also does not schedule on anything other than ASAN for try:
https://tbpl.mozilla.org/?tree=Try&rev=e1bb7915c000&showall=1

kmoir, since you're looking at bug 973900, do you mind having a look at this bug as well? I don't seem to be getting anywhere with this.

Comment 18

[Kim Moir [:kmoir] ET]

I probably can get this sorted out in a week or so. I've setup a new test master environment, just looking at the configs to see what's happening.

Comment 19

[Kim Moir [:kmoir] ET]

Attachment: bug969988.patch

So the problem is that the jittests added starting here
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1882

but then are disabled here
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#2005

because certain tests are deleted unless they reside on cedar

looking more closely, it looks like a 3/17 merge day patch wasn't removed starting here
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1953

I've tested removing the merge day stuff on my test master and the proper jittests appear.  Asking rail for comment since he was on the last merge duty.

Comment 20

[Kim Moir [:kmoir] ET]

Attachment: bug969988.patch

Talked to rail, the merge day stuff is still needed for bug 875164.  So modified the patch to just not delete the tests on try

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8398104 [details] [diff] [review]
bug969988.patch

A nit:
Can you use the following instead:
if branch in ("cedar", "try"):

Comment 22

[Kim Moir [:kmoir] ET]

Attachment: updated patch

Comment 23

[John Hopkins (:jhopkins)]

in production.

Comment 24

[Kim Moir [:kmoir] ET]

Comment on attachment 8398123 [details] [diff] [review]
updated patch

checked in before this morning's reconfig

Comment 25

[Kim Moir [:kmoir] ET]

Verified in tbpl and buildapi that tests are running/being scheduled