1509324 - Android 7.0 x86 wpt intermittent | ERROR - adb get_process_list:

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P1)

Description

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/logviewer.html#?job_id=213332123&repo=try&lineNumber=3087

[task 2018-11-22T05:56:27.410Z] 05:56:27     INFO - TEST-OK | /fetch/corb/style-html-correctly-labeled.sub.html | took 204ms
[task 2018-11-22T05:56:28.193Z] 05:56:28    ERROR - adb get_process_list: 
[task 2018-11-22T05:56:28.534Z] 05:56:28     INFO - Browser exited with return code 0

Comment 1

[Geoff Brown [:gbrown]]

There are several issues to look into here:

I think this hits wpt (I haven't seen any such failures in other test suites) because the wpt "Fennec browser" polls the returncode property, which in turn calls process_exist/get_process_list.

The error handling is suspect: Odd that no ADBError/ADBTimeoutError is reported. We could possibly surface such an error to trigger retry. (Or expand the retry list to include "adb get_process_list"?)

And why/how is get_process_list failing in the first place?

Comment 2

[Intermittent Failures Robot]

29 failures in 362 pushes (0.08 failures/push) were associated with this bug yesterday.

Repository breakdown:
* try: 29

Platform breakdown:
* android-em-7-0-x86: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1509324&startday=2018-11-23&endday=2018-11-23&tree=all

Comment 3

[Intermittent Failures Robot]

25 failures in 100 pushes (0.25 failures/push) were associated with this bug yesterday.

Repository breakdown:
* try: 25

Platform breakdown:
* android-em-7-0-x86: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1509324&startday=2018-11-24&endday=2018-11-24&tree=all

Comment 4

[Intermittent Failures Robot]

54 failures in 2387 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 54

Platform breakdown:
* android-em-7-0-x86: 54

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1509324&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #1)
> The error handling is suspect: Odd that no ADBError/ADBTimeoutError is
> reported. We could possibly surface such an error to trigger retry. (Or
> expand the retry list to include "adb get_process_list"?)

Now addressed by bug 1509614.
 
> And why/how is get_process_list failing in the first place?

It is a known problem observed for some time in bug 1466657.

Comment 6

[Geoff Brown [:gbrown]]

I hope that bug 1512686 will reduce the complexity of each check for process existence -- less adb traffic and parsing, fewer process table lookups, less opportunity for failure.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: increase poll interval - poll less often

It seems like these errors happen more frequently when we poll frequently for process existence. 0.1 seconds seems too short to me - I'll feel better if we increase it like this, to 0.5 seconds.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=20c4becb37e5999405bb11fcd885c8008cbaa122

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 9030048 [details] [diff] [review]
increase poll interval - poll less often

Worth a shot, at least.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc5f0fb59ada
Poll remote process less often during startup and shutdown; r=kwierso

Comment 10

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/cc5f0fb59ada