1317261 - Split tasks of reftests on Windows 7 debug build

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Xidorn Quan [:xidorn] UTC+10]

In bug 1258916, I'm importing a large set of tests from the CSSWG's test repo, which seems to significantly increase the time to run reftests on Windows 7 debug build, which now constantly exceeds the 7200s limit.

Is it possible to split the reftest tasks there, or extend the time limit?

(Not sure what's the right component for this issue...)

Comment 1

[Justin Wood (:Callek)]

Moving to proper component...

Comment 2

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1317261_bb.patch

The patch to split in 2 chunks the tasks of reftests on Windows 7 debug build.

Comment 3

[Xidorn Quan [:xidorn] UTC+10]

Ru is also going to timeout. Does this patch also cover that?

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: diff.txt

cat old | wc -l
14937
cat new | wc -l
14950
cat old | grep 'Windows 7' | grep ' reftest ' | grep -v 'e10s' | grep 'debug' | wc -l
13

Comment 5

[Xidorn Quan [:xidorn] UTC+10]

R, Ru, and R-e10s are all going to timeout, so they may need to be splitted as well. See https://treeherder.mozilla.org/#/jobs?repo=try&revision=ee051ed6ab12

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Win8 reftests don't run by default on Try - I think you should opt-in to those as I expect they'll need the same treatment here.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Andrei, the Try push from comment 5 also shows OSX debug around 90-100min, which is pretty high. In bug 1317434, I'm splitting both OSX and Windows debug e10s reftests into two chunks to get their runtimes down to something saner, so maybe you can just get both platforms here as well.

Comment 8

[Xidorn Quan [:xidorn] UTC+10]

Looks like Win8 debug reftests need split as well: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3b480a42bd2c4f6262c3476f7df80597c4c03cf

R-e10s takes ~109min, and R seems to exceed the time limit.

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1317261_bb.patch

This patch split in 2 chunks all tests from Windows7,Windows8 and OSX for:
reftest
reftest-noaccel
reftest_e10s 
reftest-noaccel_e10s 

Ryan I know that you also make this split for *e10-s for Windows 8,7 and OSX in bug 1317434.
So what should be the order here,first to push your patch and I will recreate this one (with changes only for reftest and reftest_noaccel on Win 7,8 as OSX tests only occur for e10-s) or should I push this patch first and modify yours.

Comment 10

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: diff

cat new | wc -l  14813
cat old | wc -l  14721

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8811721 [details] [diff] [review]
bug1317261_bb.patch

Review of attachment 8811721 [details] [diff] [review]:
-----------------------------------------------------------------

I'll rebase around yours, thanks for asking! That said, it looks like OSX only split reftest-e10s and not the non-e10s variant as well?

::: mozilla-tests/config.py
@@ +3223,5 @@
> +                 WEB_PLATFORM_REFTESTS_E10S + MOCHITEST_BC_7_E10S + MOCHITEST_CLIPBOARD_E10S + \
> +                MOCHITEST_GPU_E10S + REFTEST_NOACCEL_E10S_TWO_CHUNKS + WEB_PLATFORM_TESTS_CHUNKED_MORE_E10S
> +            BRANCHES['try']['platforms'][platform][slave_platform]['opt_unittest_suites'] += \
> +                base_tests + REFTEST_NOACCEL_E10S + WEB_PLATFORM_TESTS_CHUNKED_E10S
> +

Rather that duplicating this logic, just split Win10 too.

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1317261_bb_v1.patch

Split in 2 chunks all tests that run on  Win 7,8 and OSX for :
reftest
reftest-noaccel
reftest_e10s 
reftest-noaccel_e10s

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: diff_v1

cat new | wc -l  14813
cat old | wc -l  14721

Comment 14

[Xidorn Quan [:xidorn] UTC+10]

This may not be strictly necessary now. After experimenting some optimization, it seems the run time can be significantly reduced. In https://treeherder.mozilla.org/#/jobs?repo=try&revision=e20e45bf6c9d44c6c05a5f21600990afc7104b3b, majority of the tests now run under 100min, which gives us a large enough margin. The oranges there are from some unexpected test result (UNEXPECTED PASS?!), not timeout anymore.

The time is still much longer than Linux builds, so you may still want to apply this split, but it's no longer a blocker.

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1317261_bb_v2.patch

Comment 16

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: diff

cat old |wc -l   14721
 cat new |wc -l  14827
also spliting reftest and reftest_noaccel for OSX

Comment 17

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

https://hg.mozilla.org/build/buildbot-configs/rev/24e39a3330cd38d50260d24e3fe29e85c19caf37

Comment 18

[Ryan VanderMeulen [:RyanVM]]

In production. Initial results are also showing only a minor increase in overall runtime, so that's encouraging from an overhead and load standpoint.