1592333 - Improve raptor-browsertime page-load failure messages

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Robert Wood [:rwood]]

We should improve the failure messages when raptor-browsertime page-load tests fail. A couple of scenarios:

When browsertime fails to receive one specific type of measurement but receives the others. For example in Bug 1592048 browsertime failed to get 'firstPaint', but the resulting raptor-browsertime failure message is 14:08:06 INFO - KeyError: 'firstPaint'.

Another scenario is when browsertime fails to get any measurements at all for a single page-load iteration. The browsertime.json exists but result timings are missing for one or more iterations. For example in Bug 1592058 browsertime failed to get any measurements for one of the iterations, the final raptor-browsertime error reported is ValueError: Browsertime produced no measurements..

It would be better to make these failure errors more specific, which would be really helpful for debugging and for code sheriffs.

Comment 1

[Robert Wood [:rwood]]

Another scenario I just ran into: If browesrtime times out for some reason (i.e. times out waiting for test/script to finish) the resulting raptor-browesrtime error message is 17:44:27 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-speedometer-firefox. When really it should be more specific and indicate there was a timeout waiting for the test to finish.

Comment 2

[Robert Wood [:rwood]]

Attachment: Bug 1592333 - Improve raptor-browsertime page-load failure messages

Comment 3

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=168ed5f082ad0580b6808a921e26bb052e96b71f

Comment 4

[Robert Wood [:rwood]]

And a try push where I hacked the name of an expected measurement, to intentionally cause it not to be found in the results:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=198a9d8075053cd4038f2899c5e3fd4e799b4d60

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a08d8e9be9c2b2873d962d9d4677de492150dbfe

Comment 6

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3910e5699a6ded278fa3a323ddcb07ada2a9dfb7

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d21ccbd4d057
Improve raptor-browsertime page-load failure messages r=tarek

Comment 8

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/d21ccbd4d057