1506254 - Task retry does not work for Android 7.0 x86 wpt tasks

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Geoff Brown [:gbrown]]

Android 4.3 tasks automatically retry on ADBTimeoutError and ADBError but I haven't seen that mechanism working on Android 7.0 x86. Also, kwierso pointed out 

https://treeherder.mozilla.org/logviewer.html#?job_id=210771442&repo=try
https://treeherder.mozilla.org/logviewer.html#?job_id=210868661&repo=try

I think at least the first one should have retried.

Comment 1

[Geoff Brown [:gbrown]]

The task definition looks correct:

https://tools.taskcluster.net/groups/ctottUVpQpCq66CxrwCK0g/tasks/N0q8YhF1QzqILO88tjndbQ/details

  "onExitStatus": {
    "purgeCaches": [
      72
    ],
    "retry": [
      4,
      72
    ]
  },

Comment 2

[Geoff Brown [:gbrown]]

Works for mochitest regardless of platform:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6d8491a77cd5b8b261d5554d22e65e71391e3871

Maybe different output parser for wpt?

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #2)
> Maybe different output parser for wpt?

Yes, that seems to be the issue. The other Android test tasks use DesktopUnittestOutputParser, which checks the retry regex; Android wpt uses StructuredOutputParser, which does not check that regex.

Comment 4

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=27550465454aa8da6231495146e16222f82de545

Comment 5

[Geoff Brown [:gbrown]]

Attachment: support TBPL_RETRY in structured logging output parser

The DesktopUnittestOutputParser already supports TBPL_RETRY. There's no convenient way to move support to the base class which doesn't know about self.tbpl_status and is currently free of higher level mozharness dependencies.

In addition to detecting retry conditions and setting tbpl_status, we need to be careful not to clobber TBPL_RETRY when doing the summary processing required for TV and similar runs.

Here's an example with retry forced by an ADBError:
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0009814cdd5e1401f9b7a2f74fe9027730940920

and another try run to check that normal success/failure status is generally okay:
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5ad175d3d559319c6a7624eaf2b638f89be290a1

Comment 6

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 9026576 [details] [diff] [review]
support TBPL_RETRY in structured logging output parser

Review of attachment 9026576 [details] [diff] [review]:
-----------------------------------------------------------------

we only have to update structuredlog.py ?

Comment 7

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #6)
> we only have to update structuredlog.py ?

Yes, I think so.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/09abaef41d5e
Support TBPL_RETRY in structured logger, to enable task retry; r=jmaher

Comment 9

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/09abaef41d5e

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Should this have made the `ERROR - adb get_process_list:` failures trigger a retry, or was comment 0 just kinda hoping it might have been fixed along with the `ls could not be found` case?

Because I rebased onto the merge to m-c that included this patch and I'm still getting failures like https://treeherder.mozilla.org/logviewer.html#?job_id=213332123&repo=try that aren't retrying.

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

And this one hit the get_process_list error twice in under a second: https://treeherder.mozilla.org/logviewer.html#?job_id=213332149&repo=try&lineNumber=2532

Comment 12

[Geoff Brown [:gbrown]]

Retry will be triggered by "ADBError" or "ADBTimeoutError" in the log, but not this strange get_process_list error -- I filed bug 1509324 for that and will follow up.

Comment 13

[Intermittent Failures Robot]

12 failures in 2387 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 12

Platform breakdown:
* android-em-7-0-x86: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1506254&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 14

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/17c90b5fed5d