revisit chunking of web-platform-tests-reftests

RESOLVED FIXED in Firefox 68

Status

enhancement
P2
normal
RESOLVED FIXED
2 months ago
Last month

People

(Reporter: egao, Assigned: egao)

Tracking

(Blocks 1 bug)

unspecified
mozilla68
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(1 attachment)

Assignee

Description

2 months ago

Summary

I noticed that for most of the platforms, web-platform-tests-reftests currently running on mozilla-central, the chunks are set to 6.

This produces tests that run for a relatively short duration in multiple chunks, as exemplified in this try push:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=pending%2Crunning%2Csuperseded%2Cusercancel%2Cretry%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&tier=1%2C2%2C3&group_state=expanded&revision=e8aebe488b2f2e567940577de25013d00e818f7c&searchStr=wr

Data

Using Wr1 as standardized example, in the mozilla-central revisio e8aebe488b2f2e567940577de25013d00e818f7c (linked above):

linux64-shippable/opt: 13 minutes, 17:58:49 - 17:54:45 = 00:04:04 spent on setup
windows10-64-shippable/opt: 11 minutes, 18:10:29 - 18:06:32 = 00:03:57 spent on setup
macosx64-shippable/opt: 13 minutes, 17:34:48 - 17:32:15 = 00:02:33 spent on setup

Thoughts

Perhaps investigate if it is feasible to reduce chunks for common platforms that have runtimes of around 20 minutes or less, such that each chunk takes approximately 30 minutes to execute (which is an informal line set in discussions, that chunks ideally should not exceed 30 minutes)

Assignee

Updated

2 months ago
Assignee: nobody → egao
Assignee

Comment 1

2 months ago

I have two try runs with some changes:

4 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b20d25df4f5bd52bae7a8183d580c1772e973b4f

Only a small handful of tests were run in this push:

  • windows10-64
  • windows10-aarch64

windows10-aarch64 has a failure, not sure if this is caused by the chunk resizing.

Overall, this led me to believe I could further reduce the chunks for faster, more stable platforms (eg. windows10-64, linux64)

3 chunks across the board

https://treeherder.mozilla.org/#/jobs?repo=try&revision=68f10e20aef5f89bf7598f5165c326e25d06d304

All platforms were run, and some failures were noted.

It was also noted that ccov builds do not like being put into small chunks since their execution time is long.

Chunk numbers adjusted per platform

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da081f84279ecd619f503bcebef59cfeb3495f4a

Chunks were adjusted as follows:

windows10-aarch64/.*: 4
windows7-32/.*: 4
linux64-ccov/debug: 6
macosx64.*/.*: 6
android.*: 6
default: 3
Assignee

Updated

2 months ago
Type: defect → enhancement
Assignee

Updated

2 months ago
Summary: reduce chunks for web-platform-tests-reftests → revisit chunking of web-platform-tests-reftests
Assignee

Updated

2 months ago
Blocks: 1548160
Priority: -- → P2
Assignee

Updated

2 months ago
No longer blocks: 1548160
Assignee

Updated

2 months ago
Blocks: 1548924
Assignee

Updated

2 months ago
No longer blocks: 1548924
Attachment #9062570 - Attachment description: Bug 1548106 - revisit chunking of web-platform-tests-reftests → Bug 1548106 - task efficiency: review chunking of web-platform-tests-reftests
Assignee

Updated

2 months ago
Blocks: 1548965
Assignee

Updated

2 months ago
No longer blocks: 1548965
Assignee

Updated

Last month
Assignee

Updated

Last month

Comment 3

Last month
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/427bae2148d6
task efficiency: review chunking of web-platform-tests-reftests r=jmaher,gbrown

Comment 4

Last month
bugherder
Status: NEW → RESOLVED
Closed: Last month
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.