1623027 - Improve error line for page load failures

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

To better investigate the page load errors via bug 1614898 it would be good to have a better error line in the log output for classification. Right now it shows:

[task 2020-03-15T11:39:35.011Z] 11:39:09 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-tp6m-jianshu-geckoview-cold' timed out loading test page: https://www.jianshu.com/ pending metrics: fcp, fnb paint

We started to discuss that on bug 1614898 comment 5 and bug 1614898 comment 8, but lets move the discussion over here.

What I would propose is:

[task 2020-03-15T11:39:35.011Z] 11:39:09 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-tp6m-jianshu-geckoview-cold' timed out loading test page

This would allow us to have generic bugs for each test, which doesn't cause multiple bugs per pending metric. To further include the details we could have a warning line right before, like:

[task 2020-03-15T11:39:35.011Z] 11:39:09 WARN - raptor-main WARN: cycle 1 has pending metrics for: fcp, fnb paint

What do you all think about? Florin, feel free to also discuss this with your team (so I don't have to add everyone as needinfo here). Thanks.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This proposal sounds good to me.

Comment 2

[Florin Strugariu [:Bebe]]

I agree with the proposal.
Having all the errors of the same test in one bug is a smart move!

Comment 3

[Greg Mierzwinski [:sparky]]

+1 for removing the pending metrics.

Although, could we still leave a mention of the pending metrics in the timeout log? This way the error message will still have a mention about the pending metrics without worrying about what they are. We should still have the pending metrics WARN log that you mentioned though.
[task 2020-03-15T11:39:35.011Z] 11:39:09 ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: test 'raptor-tp6m-jianshu-geckoview-cold' timed out loading test page while waiting for pending metrics

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Florin, could someone from your team can take care of that? I won't have the time right now. Thanks.

Comment 5

[Marian Raiciof [:marauder]]

Attachment: Bug 1623027 - Improve error line for page load failures

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/53f15b782246
Improve error line for page load failures r=perftest-reviewers,AlexandruIonescu,whimboo,sparky

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/53f15b782246