1478775 - Mark raptor test failed if any one test page times out

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Robert Wood [:rwood]]

I noticed that if a single test page in a raptor test times out, but the other pages work fine, the timeout isn't reported. The test results will just have the results for the successful pages but the page that timed out is omitted.

If one test page times out then the entire test should be marked as a failure, otherwise we may not notice in production that results for one of the pages is missing.

i.e. If running raptor-tp6, and any one of the pages within times out, the tp6 test job should be marked as a failure.

Comment 1

[Robert Wood [:rwood]]

If for example one of the tp6 test pages times out, but the others are successful, the results for the successful test pages will still be dumped in PERFHERDER_DATA however the overall test job will be marked as a failure (so a failed pageload won't be missed on taskcluster).

Comment 2

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7195f7662bdb8b529c5ae3ebadfa46581cbcb682

Comment 3

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=70454b6a589993bc1f902c9a9882ecdc20047552

Comment 4

[Robert Wood [:rwood]]

Attachment: Bug 1478775 - Mark raptor test failed if any one test page times out

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec5f256bd5204df841e5c9c8c6251910683e0e17

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9007888 [details]
Bug 1478775 - Mark raptor test failed if any one test page times out

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 7

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e39db385bd21a09bb65a97b3320c1cf222e9b62a

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b2dd48a11705
Mark raptor test failed if any one test page times out r=jmaher

Comment 9

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/b2dd48a11705