1548106 - revisit chunking of web-platform-tests-reftests

Status: RESOLVED FIXED

(Testing :: General, enhancement, P2)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Summary

I noticed that for most of the platforms, web-platform-tests-reftests currently running on mozilla-central, the chunks are set to 6.

This produces tests that run for a relatively short duration in multiple chunks, as exemplified in this try push:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=pending%2Crunning%2Csuperseded%2Cusercancel%2Cretry%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&tier=1%2C2%2C3&group_state=expanded&revision=e8aebe488b2f2e567940577de25013d00e818f7c&searchStr=wr

Data

Using Wr1 as standardized example, in the mozilla-central revisio e8aebe488b2f2e567940577de25013d00e818f7c (linked above):

linux64-shippable/opt: 13 minutes, 17:58:49 - 17:54:45 = 00:04:04 spent on setup
windows10-64-shippable/opt: 11 minutes, 18:10:29 - 18:06:32 = 00:03:57 spent on setup
macosx64-shippable/opt: 13 minutes, 17:34:48 - 17:32:15 = 00:02:33 spent on setup

Thoughts

Perhaps investigate if it is feasible to reduce chunks for common platforms that have runtimes of around 20 minutes or less, such that each chunk takes approximately 30 minutes to execute (which is an informal line set in discussions, that chunks ideally should not exceed 30 minutes)

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

I have two try runs with some changes:

4 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b20d25df4f5bd52bae7a8183d580c1772e973b4f

Only a small handful of tests were run in this push:

• windows10-64
• windows10-aarch64

windows10-aarch64 has a failure, not sure if this is caused by the chunk resizing.

Overall, this led me to believe I could further reduce the chunks for faster, more stable platforms (eg. windows10-64, linux64)

3 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=68f10e20aef5f89bf7598f5165c326e25d06d304

All platforms were run, and some failures were noted.

It was also noted that ccov builds do not like being put into small chunks since their execution time is long.

Chunk numbers adjusted per platform

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da081f84279ecd619f503bcebef59cfeb3495f4a

Chunks were adjusted as follows:

windows10-aarch64/.*: 4
windows7-32/.*: 4
linux64-ccov/debug: 6
macosx64.*/.*: 6
android.*: 6
default: 3

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1548106 - task efficiency: review chunking of web-platform-tests-reftests

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/427bae2148d6
task efficiency: review chunking of web-platform-tests-reftests r=jmaher,gbrown

Comment 4

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/427bae2148d6