Closed Bug 1643177 Opened 1 year ago Closed 1 year ago

Reduce web-platform-tests-backlog timeout multiplier

Categories

(Testing :: General, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: egao, Assigned: egao)

References

Details

Attachments

(7 files, 1 obsolete file)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

:jmaher has moved a lot of tests from web-platform-tests and web-platform-tests-reftest to the backlog side of things in the parent bug.

Tests for features that are not implemented yet, or will not be implemented, are currently run as separate suites.

Since the tests in the backlog side of things are expected to time out by default, let's find a way to safely reduce the time the test harness waits for the timeout result. This would help us reduce the runtime of the tests, which in turn will help us reduce the cost of machine time.

If things go as planned, it may even be possible to run all backlog tests in one chunk, therefore saving us machine time and setup/teardown overhead of multiple chunks.

Assignee: nobody → egao

Changes:

  • due to lower timeout multiplier, some subtests do not have the chance to run. These expectations are updated with the possibility that tests will TIMEOUT or NOTRUN.
Attachment #9156070 - Attachment description: Bug 1643177 - web-platform-tests-backlog to set timeout-multiplier of 0.4 → Bug 1643177 - web-platform-tests-backlog to set timeout-multiplier of 0.25
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9931a8bb043c
adjust test expectations of some web-platform-tests-backlog tests r=jmaher
Attachment #9157036 - Attachment description: Bug 1643177 - discontinue web-platform-tests-backlog on ccov platforms → Bug 1643177 - discontinue web-platform-tests-backlog on ccov/devedition platforms
Attachment #9157036 - Attachment description: Bug 1643177 - discontinue web-platform-tests-backlog on ccov/devedition platforms → Bug 1643177 - discontinue backlog-type web-platform-tests on ccov/devedition platforms
Attachment #9157036 - Attachment description: Bug 1643177 - discontinue backlog-type web-platform-tests on ccov/devedition platforms → Bug 1643177 - discontinue web-platform-tests-backlog on ccov/devedition platforms
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b5133957b5e
discontinue web-platform-tests-backlog on ccov/devedition platforms r=jmaher
Status: NEW → ASSIGNED
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3acd697a9131
web-platform-tests-backlog to set timeout-multiplier of 0.25 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/745ca9baa9c8
reduce web-platform-tests-backlog chunk counts r=jmaher
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/024c23596ff6
web-platform-tests-backlog to set timeout-multiplier of 0.25 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/1793ec502846
reduce web-platform-tests-backlog chunk counts r=jmaher

Added additional conditional to check if the test_type is testharness.

Flags: needinfo?(egao)

Comment on attachment 9157684 [details]
Bug 1643177 - add TIMEOUT expectations for some tests in network-error-logging for macosx1014 web-platform-tests-backlog

Revision D80219 was moved to bug 1646319. Setting attachment 9157684 [details] to obsolete.

Attachment #9157684 - Attachment is obsolete: true

Bug 1646319 is for linux1804-64/debug failures due to this change.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5065d048c930
add expectations for some /network-error-logging tests for web-platform-tests-backlog on linux1804-64/debug r=jmaher
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/579780116dbf
apply timeout multiplier reduction to web-platform-tests-reftest-backlog and reduce chunks r=jmaher
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4eab38aba3fe
separate web-platform-tests-backlog tests into its own test-set r=jmaher

Multiplier has been reduced to 0.25. Further tuning should be done in its own bug.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Keywords: leave-open
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.