1128543 - Break reftests into two chunks on linux64 debug

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nicolas Silva [:nical]]

OMTC on Linux makes reftests run significantly slower, which makes us hit the 7200 seconds limit for debug 64bit builds (32bit being already split we don't hit the limit).
Interestingly, talos performance isn't affected like this, OMTC linux is actually nice for performance in most cases (especially if you watch videos on the web). So far, my investigations have pointed to X11 inefficiencies with reading back the content (which reftests do a lot) probably specific to driver versions since I can't reproduce the issue on my own computers.
We can't get rid of the X11/xrender drawing in the short term because of a long list of dependencies, but we want OMTC on linux soon.

I propose that we bite the bullet and split linux64 debug reftests in two so that we can land OMTC now, until we have finally removed our dependency to xrender.

Comment 1

[Nicolas Silva [:nical]]

Attachment: Split reftests into two chunks on linux 64 for gecko 39+

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8573224 [details] [diff] [review]
Split reftests into two chunks on linux 64 for gecko 39+

Review of attachment 8573224 [details] [diff] [review]:
-----------------------------------------------------------------

wow, you're brave touching config.py !!

This should get you what you want and I'll r+ this so you have the choice to land however I would recommend that you land this slowly like on a single project branch (cedar is our testing playground) first and then all of trunk. If there is bustage, this will require time from sheriffs and releng to backout and do another reconfig of buildbot. Since this is untested, there are a number of places this could go wrong: from the mozharness script to the actual test harness that mozharness calls. Granted we have already successfully split reftests with linux32, there could be hard coded bits specific to arch.

If you have tested this with mozharness already either locally on your machine or by doing a refest try push and messing with the mozharness repo to force a chunk, then we should just land this across all of trunk at once.

I see you found jgriffin's patch to land linux32 reftest split across 36: http://hg.mozilla.org/build/buildbot-configs/rev/847725a71b7e
for context, here was his initial patch testing on cedar: http://hg.mozilla.org/build/buildbot-configs/diff/4cefbd4f12be/mozilla-tests/config.py

::: mozilla-tests/config.py
@@ +2072,5 @@
> +# reftest is chunked on linux64 for gecko >= 39
> +for platform in PLATFORMS.keys():
> +    for name, branch in items_at_least(BRANCHES, 'gecko_version', 39):
> +        for slave_platform in PLATFORMS[platform]['slave_platforms']:
> +            if slave_platform not in ['ubuntu64_vm']:

parsing the top of this file for what platform will have 'ubuntu64_vm' I see:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py?rev=2bcf41d107df#176
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py?rev=2bcf41d107df#201

but I also see we have a different name associated with asan:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py?rev=2bcf41d107df#190

It looks like we also do reftests for linux64 asan builds. did you want to split those too?

Comment 3

[Nicolas Silva [:nical]]

Attachment: Split reftests into two chunks on linux 64 debug for gecko 39+

Hi Jordan, thanks for the detailed review (very useful as I am discovering this stuff)!
On second thought, I think that it's sufficient for now do this with debug targets only (which leaves out asan build). Here's the updated patch.

Comment 4

[Nicolas Silva [:nical]]

Attachment: Cedar patch

Following jgriffin's example, here's the patch to try it out on cedar first.

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8573897 [details] [diff] [review]
Cedar patch

Review of attachment 8573897 [details] [diff] [review]:
-----------------------------------------------------------------

cool :)

I think we almost have it.

r+ if you adjust your patch to support in line comments.

::: mozilla-tests/config.py
@@ +1875,5 @@
>  BRANCHES['try']['platforms']['macosx64']['yosemite']['opt_unittest_suites'] = UNITTEST_SUITES['opt_unittest_suites'][:]
>  BRANCHES['try']['platforms']['macosx64']['yosemite']['debug_unittest_suites'] = UNITTEST_SUITES['debug_unittest_suites'][:]
>  
>  ######## cedar
> +BRANCHES['cedar']['platforms']['linux']['ubuntu64_vm']['debug_unittest_suites'] = \

it's confusing but 'linux' represents linux 32 arch while 'linux64' is for 64 arch.

iow - we want s/linux/linux64 here.

@@ +1876,5 @@
>  BRANCHES['try']['platforms']['macosx64']['yosemite']['debug_unittest_suites'] = UNITTEST_SUITES['debug_unittest_suites'][:]
>  
>  ######## cedar
> +BRANCHES['cedar']['platforms']['linux']['ubuntu64_vm']['debug_unittest_suites'] = \
> +    [x for x in BRANCHES['cedar']['platforms']['linux']['ubuntu64_vm']['debug_unittest_suites'] if x[0] != 'reftest'] + REFTEST_TWO_CHUNKS[:]

s/linux/linux64 here too

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8573896 [details] [diff] [review]
Split reftests into two chunks on linux 64 debug for gecko 39+

Review of attachment 8573896 [details] [diff] [review]:
-----------------------------------------------------------------

should work :)

let's land this after we land and try out the cedar patch

Comment 7

[Nicolas Silva [:nical]]

Updated the cedar patch and landed it: https://hg.mozilla.org/build/buildbot-configs/rev/7b7711ab6689

Comment 8

[Nicolas Silva [:nical]]

Cedar is looking good.

Comment 9

[Nicolas Silva [:nical]]

Backed out the cedar-specific patch and landed the real one.

https://hg.mozilla.org/build/buildbot-configs/rev/fc52aa0ed8eb
https://hg.mozilla.org/build/buildbot-configs/rev/d66f673350a3

Comment 10

[Jordan Lund (:jlund)]

(In reply to Nicolas Silva [:nical] from comment #9)
> Backed out the cedar-specific patch and landed the real one.
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/fc52aa0ed8eb
> https://hg.mozilla.org/build/buildbot-configs/rev/d66f673350a3

\o/

Comment 11

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/fc52aa0ed8eb

Comment 12

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d66f673350a3