1643177 - Reduce web-platform-tests-backlog timeout multiplier

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

:jmaher has moved a lot of tests from web-platform-tests and web-platform-tests-reftest to the backlog side of things in the parent bug.

Tests for features that are not implemented yet, or will not be implemented, are currently run as separate suites.

Since the tests in the backlog side of things are expected to time out by default, let's find a way to safely reduce the time the test harness waits for the timeout result. This would help us reduce the runtime of the tests, which in turn will help us reduce the cost of machine time.

If things go as planned, it may even be possible to run all backlog tests in one chunk, therefore saving us machine time and setup/teardown overhead of multiple chunks.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Possible reference: https://phabricator.services.mozilla.com/D31176, https://phabricator.services.mozilla.com/D25115

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - web-platform-tests-backlog to set timeout-multiplier of 0.25

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - reduce web-platform-tests-backlog chunk counts

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - adjust test expectations of some web-platform-tests-backlog tests

Changes:

• due to lower timeout multiplier, some subtests do not have the chance to run. These expectations are updated with the possibility that tests will TIMEOUT or NOTRUN.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9931a8bb043c
adjust test expectations of some web-platform-tests-backlog tests r=jmaher

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - discontinue web-platform-tests-backlog on ccov/devedition platforms

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b5133957b5e
discontinue web-platform-tests-backlog on ccov/devedition platforms r=jmaher

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3acd697a9131
web-platform-tests-backlog to set timeout-multiplier of 0.25 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/745ca9baa9c8
reduce web-platform-tests-backlog chunk counts r=jmaher

Comment 9

[Razvan Maries]

Backed out for multiple perma failures.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=a1P9Y8prQr-Vfb1z5s0CrA.0&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=wd3&revision=745ca9baa9c8891decb1394aca0b178299fd6d29

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306554537&repo=autoland&lineNumber=6014
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306550797&repo=autoland&lineNumber=6067
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306550801&repo=autoland&lineNumber=1227

Backout: https://hg.mozilla.org/integration/autoland/rev/575059fe05879f2dd0ca0dac7d522d280391c3fc

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/024c23596ff6
web-platform-tests-backlog to set timeout-multiplier of 0.25 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/1793ec502846
reduce web-platform-tests-backlog chunk counts r=jmaher

Comment 11

[Edwin Takahashi (:egao | infrequent contributor)]

Added additional conditional to check if the test_type is testharness.

Comment 12

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/024c23596ff6
https://hg.mozilla.org/mozilla-central/rev/1793ec502846

Comment 13

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - add TIMEOUT expectations for some tests in network-error-logging for macosx1014 web-platform-tests-backlog

Comment 14

[Phabricator Automation]

Comment on attachment 9157684 [details]
Bug 1643177 - add TIMEOUT expectations for some tests in network-error-logging for macosx1014 web-platform-tests-backlog

Revision D80219 was moved to bug 1646319. Setting attachment 9157684 [details] to obsolete.

Comment 15

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - add expectations for some /network-error-logging tests for web-platform-tests-backlog on linux1804-64/debug

Comment 16

[Edwin Takahashi (:egao | infrequent contributor)]

Bug 1646319 is for linux1804-64/debug failures due to this change.

Comment 17

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - apply timeout multiplier reduction to web-platform-tests-reftest-backlog and reduce chunks

Comment 18

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1643177 - separate web-platform-tests-backlog tests into its own test-set

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5065d048c930
add expectations for some /network-error-logging tests for web-platform-tests-backlog on linux1804-64/debug r=jmaher

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/579780116dbf
apply timeout multiplier reduction to web-platform-tests-reftest-backlog and reduce chunks r=jmaher

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4eab38aba3fe
separate web-platform-tests-backlog tests into its own test-set r=jmaher

Comment 22

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/5065d048c930
https://hg.mozilla.org/mozilla-central/rev/579780116dbf

Comment 23

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/4eab38aba3fe

Comment 24

[Edwin Takahashi (:egao | infrequent contributor)]

Multiplier has been reduced to 0.25. Further tuning should be done in its own bug.