1637499 - Perma Android 7.0 MotoG5 Rap-refbrow tp6m-c-1 | [taskcluster:error] Aborting task...

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The tp6m-c-1 job for refbrow on MotoG5 is perma-failing on mozilla-central since May 8th:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedTaskRun=Uoen0Sf2RxmIVuvc9vAkkQ-0&searchStr=rap%2Cmotog5%2Crefbrow&fromchange=f9356df325cd5df64ef952636b788db27fbfd19f

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302041751&repo=mozilla-central&lineNumber=5469

As it looks like the task duration doesn't seem to be enough to get all the tests run within 30 minutes.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I asked the sheriffs for a backfill. Lets see what it is. Maybe it's related to the backout of the mitmproxy 5.0.1 update on bug 1624813.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It indeed regressed by the mitmproxy 5.0.1 backout:

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2C7.0%2Cmotog5%2Cpgo%2Craptor%2Cperformance%2Ctests%2Con%2Cthe%2Creference%2Cbrowser%2Ctest-android-hw-g5-7-0-arm7-api-16%2Fpgo-raptor-tp6m-1-refbrow-cold-e10s%2Crap-refbrow%28tp6m-c-1%29&tochange=c01c4a6969702444a68d28fe4f3057f8532770b8&fromchange=18e5112a6745016bc8546ca4ede94225af9e5dbd

Which means the upgrade caused this failure to stop a while ago, and now it is back. With 5.0.1 the jobs finished within 17 minutes and now we timeout after 30 minutes! This is nearly twice that long.

It would be good if someone could compare the timings between a passing and failing job within the logs. Maybe that already gives us an idea what's causing it.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Florin, given that you set P1 I assume you are going to work on this bug?

Comment 4

[Intermittent Failures Robot]

7 failures in 5488 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1637499&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 5

[Florin Strugariu [:Bebe]]

my mistake here I should have added P2 as Priority so this can be picked up later.
Not currently working on it

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There is nothing we have to take care about it anymore here, given that the offending patch (mitmproxy 5.0.1 upgrade) has been backed-out. We only have to make sure to not regress that part again.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually it's still happening but gets inappropriately classified for bug 1502032 due to the missing intermittent-failure keyword on this bug:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302552078&repo=mozilla-central&lineNumber=5506

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Florin was the 301 patch also backed out? Could that one have caused this problem?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It looks like bug 1628627 hasn't been backed out. I assume we should also do that?

Comment 10

[Florin Strugariu [:Bebe]]

Yes you are right Bug 1628627 - [mozproxy] Mitmproxy is ignoring 301 requests should be also backed out

Comment 11

[Florin Strugariu [:Bebe]]

added a request in https://bugzilla.mozilla.org/show_bug.cgi?id=1628627#c5

Comment 12

[Intermittent Failures Robot]

2 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1637499&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Marking as fixed by the backout of bug 1628627.