Closed Bug 1592333 Opened 1 year ago Closed 11 months ago

Improve raptor-browsertime page-load failure messages

Categories

(Testing :: Raptor, enhancement, P1)

Version 3
enhancement

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

(Blocks 2 open bugs)

Details

Attachments

(1 file)

We should improve the failure messages when raptor-browsertime page-load tests fail. A couple of scenarios:

When browsertime fails to receive one specific type of measurement but receives the others. For example in Bug 1592048 browsertime failed to get 'firstPaint', but the resulting raptor-browsertime failure message is 14:08:06 INFO - KeyError: 'firstPaint'.

Another scenario is when browsertime fails to get any measurements at all for a single page-load iteration. The browsertime.json exists but result timings are missing for one or more iterations. For example in Bug 1592058 browsertime failed to get any measurements for one of the iterations, the final raptor-browsertime error reported is ValueError: Browsertime produced no measurements..

It would be better to make these failure errors more specific, which would be really helpful for debugging and for code sheriffs.

Another scenario I just ran into: If browesrtime times out for some reason (i.e. times out waiting for test/script to finish) the resulting raptor-browesrtime error message is 17:44:27 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-speedometer-firefox. When really it should be more specific and indicate there was a timeout waiting for the test to finish.

Assignee: nobody → rwood
Status: NEW → ASSIGNED
Priority: P2 → P1
Priority: P1 → P2
Priority: P2 → P1

And a try push where I hacked the name of an expected measurement, to intentionally cause it not to be found in the results:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=198a9d8075053cd4038f2899c5e3fd4e799b4d60

Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d21ccbd4d057
Improve raptor-browsertime page-load failure messages r=tarek
Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
You need to log in before you can comment on or make changes to this bug.