Improve Raptor no raptor test results were found

RESOLVED FIXED in Firefox 67

Status

enhancement
P3
normal
RESOLVED FIXED
6 months ago
5 months ago

People

(Reporter: bc, Assigned: bc)

Tracking

(Blocks 1 bug)

Trunk
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox67 fixed)

Details

(Whiteboard: [cia:logs:2019])

Attachments

(1 attachment)

Currently we are tracking a large number of Raptor failures in bug 1499253 but do not have the ability to drill down into the separate issues due to the generic nature of the failure message:

21:52:32 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found

This does not distinguish failures on different tests which may have different root causes. For example, this does not distinguish Unity Webgl (ugl) from Speedometer (sp) failures which have very different failure rates nor does it track the app which is being tested.

Since raptor could run multiple tests in a single invocation, we need to be able to track which individual test failed rather than a single overall message.

Thanks :bc, yes this can definitely be improved. For page-load tests it does report which specific page(s) timed out, at the bottom of the overall suite run. I'm sure that can be made clearer though, and ensure the same is done for all types of tests.

Priority: -- → P3

Example output:

raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-speedometer-geckoview

This will allow sheriffs to distinguish the failures of different raptor tests and will enable us to focus on the particular tests which are the most problematic.

Aryx: This will mean that bug 1499253 will no longer be the go to bug for raptor failures and that we will potentially get a separate bug for each raptor test. Is that ok?

Attachment #9040274 - Flags: review?(rwood)
Attachment #9040274 - Flags: feedback?(aryx.bugmail)
Attachment #9040274 - Flags: feedback?(aryx.bugmail) → feedback+
Comment on attachment 9040274 [details] [diff] [review]
bug-1520130-raptor-no-test-results.patch

Review of attachment 9040274 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM and tried it out locally. Thanks!
Attachment #9040274 - Flags: review?(rwood) → review+
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9287ec4e4115
[raptor] Improve Raptor no raptor test results were found message, r=rwood.
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Depends on: 1524495
See Also: → 1525399
You need to log in before you can comment on or make changes to this bug.