Closed Bug 1548106 Opened 1 year ago Closed 1 year ago

revisit chunking of web-platform-tests-reftests

Categories

(Testing :: General, enhancement, P2)

enhancement

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: egao, Assigned: egao)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Summary

I noticed that for most of the platforms, web-platform-tests-reftests currently running on mozilla-central, the chunks are set to 6.

This produces tests that run for a relatively short duration in multiple chunks, as exemplified in this try push:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=pending%2Crunning%2Csuperseded%2Cusercancel%2Cretry%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&tier=1%2C2%2C3&group_state=expanded&revision=e8aebe488b2f2e567940577de25013d00e818f7c&searchStr=wr

Data

Using Wr1 as standardized example, in the mozilla-central revisio e8aebe488b2f2e567940577de25013d00e818f7c (linked above):

linux64-shippable/opt: 13 minutes, 17:58:49 - 17:54:45 = 00:04:04 spent on setup
windows10-64-shippable/opt: 11 minutes, 18:10:29 - 18:06:32 = 00:03:57 spent on setup
macosx64-shippable/opt: 13 minutes, 17:34:48 - 17:32:15 = 00:02:33 spent on setup

Thoughts

Perhaps investigate if it is feasible to reduce chunks for common platforms that have runtimes of around 20 minutes or less, such that each chunk takes approximately 30 minutes to execute (which is an informal line set in discussions, that chunks ideally should not exceed 30 minutes)

Assignee: nobody → egao

I have two try runs with some changes:

4 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b20d25df4f5bd52bae7a8183d580c1772e973b4f

Only a small handful of tests were run in this push:

  • windows10-64
  • windows10-aarch64

windows10-aarch64 has a failure, not sure if this is caused by the chunk resizing.

Overall, this led me to believe I could further reduce the chunks for faster, more stable platforms (eg. windows10-64, linux64)

3 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=68f10e20aef5f89bf7598f5165c326e25d06d304

All platforms were run, and some failures were noted.

It was also noted that ccov builds do not like being put into small chunks since their execution time is long.

Chunk numbers adjusted per platform

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da081f84279ecd619f503bcebef59cfeb3495f4a

Chunks were adjusted as follows:

windows10-aarch64/.*: 4
windows7-32/.*: 4
linux64-ccov/debug: 6
macosx64.*/.*: 6
android.*: 6
default: 3
Type: defect → enhancement
Summary: reduce chunks for web-platform-tests-reftests → revisit chunking of web-platform-tests-reftests
Blocks: 1548160
Priority: -- → P2
No longer blocks: 1548160
Blocks: 1548924
No longer blocks: 1548924
Attachment #9062570 - Attachment description: Bug 1548106 - revisit chunking of web-platform-tests-reftests → Bug 1548106 - task efficiency: review chunking of web-platform-tests-reftests
Blocks: 1548965
No longer blocks: 1548965
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/427bae2148d6
task efficiency: review chunking of web-platform-tests-reftests r=jmaher,gbrown
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.