1319196 - Timeout exceeded for shell call 'adb shell dumpsys window windows; echo $?'

Status: RESOLVED DUPLICATE of bug 1412297

(Firefox for Android Graveyard :: Testing, defect, P3)

Description

[Geoff Brown [:gbrown]]

Our Android browser test harnesses (for mochitest, reftest, and robocop) use dumpsys to check that Fennec is in the foreground. Recently (last week, two weeks ago?) we started getting low-frequency intermittent test failures where the dumpsys call timed out, causing the test harness to abort the tests.

I'll try to dup related bugs here.

I have also been seeing these failures regularly in experimental mochitest-chrome runs on x86 - bug 1315749. Those failures appear to go away with the fix for bug 1318245, so I'm hoping this bug will dup to 1318245.


For x86 mochitest chrome compare:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76fc3f25d11b5586cd076f703c6565de1cdbbe95

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d13432af39a719f89edae42be69cc9aae9b897f

Comment 2

[Geoff Brown [:gbrown]]

Spotted these recent failures, for x86 mochitest chrome:

https://treeherder.mozilla.org/logviewer.html#?job_id=39815072&repo=mozilla-inbound#L3768
https://treeherder.mozilla.org/logviewer.html#?job_id=39765106&repo=mozilla-inbound#L3703

Comment 6

[Geoff Brown [:gbrown]]

I notice "unable to restore focus" in many of the recent reports, just prior to the dumpsys timeout.

Comment 7

[Geoff Brown [:gbrown]]

...typically following test_session_zombification.html in mochitest-chrome.

Comment 13

[Geoff Brown [:gbrown]]

test_session_zombification.html issue was likely addressed by bug 1320987, but other similar failures persist. I'm quite puzzled -- I can't find or think of any android harness or environment changes that might account for this new type of failure.

Comment 14

[Geoff Brown [:gbrown]]

Attachment: increase dumpsys timeout

I wonder if increasing the timeout will help. If it doesn't help, I'll back this out.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8818112 [details] [diff] [review]
increase dumpsys timeout

Review of attachment 8818112 [details] [diff] [review]:
-----------------------------------------------------------------

can you add a comment to indicate why you chose 60?

Comment 16

[Geoff Brown [:gbrown]]

Sure: "Increased timeout to 60 seconds following intermittent timeouts at 30."

Comment 17

[Geoff Brown [:gbrown]]

Attachment: try to get a screenshot

I'd also like to capture the screen when this happens, in case that shows something interesting. Performing the screenshot in kill() should also provide a capture on timeouts or other abnormal conditions resulting in the harness forcing the process down.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8818314 [details] [diff] [review]
try to get a screenshot

Review of attachment 8818314 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c902a14dffee
Increase timeout for 'adb shell dumpsys'; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/f21a93061f9a
Take screenshot when dumpsys fails; r=jmaher

Comment 20

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/c902a14dffee
https://hg.mozilla.org/mozilla-central/rev/f21a93061f9a

Comment 25

[Geoff Brown [:gbrown]]

Screenshots are working great, but don't show anything out of the ordinary.

I don't think the timeout increase was effective.

Comment 26

[Geoff Brown [:gbrown]]

Recent failures (today's dups, above) are all Android x86 mochitest-chrome; we just started running that job a few weeks ago.

In those logs, the anr report shows dumpsys running and blocking another window manager thread (if I'm reading that right).

Comment 35

[Geoff Brown [:gbrown]]

Attachment: call dumpsys less often

We can call dumpsys less often, by calling getTopActivity() less - only when the harness has not received log output recently. Call it less, hopefully it will fail less!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36de8eb4c9c962b0022e97136fd7c7855cce7bfe

Comment 36

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8824582 [details] [diff] [review]
call dumpsys less often

Review of attachment 8824582 [details] [diff] [review]:
-----------------------------------------------------------------

nice!

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f02e4baff85d
Call dumpsys less during Android tests; r=jmaher

Comment 38

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/f02e4baff85d

Comment 42

[Geoff Brown [:gbrown]]

It looks like comment 38 helped reduce the frequency of this problem quite a lot; since landing, I've only seen bug 1331642. Next step here might be ignoring dumpsys failures and retrying, cautiously.

Comment 44

[Geoff Brown [:gbrown]]

Bug 1335501 began triggering job retry for a very similar, but slightly different, error message. We can consider the same approach here, if needed.

Comment 45

[Geoff Brown [:gbrown]]

I think this problem is still possible. I want to keep the bug open in case test failures return. However, I'm not going to work on this otherwise.

Comment 46

[Intermittent Failures Robot]

2 failures in 924 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* android-4-2-x86: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1319196&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 47

[Intermittent Failures Robot]

1 failures in 864 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1319196&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 48

[Intermittent Failures Robot]

1 failures in 912 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-4-3-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1319196&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 49

[Geoff Brown [:gbrown]]

Bug 1412297 will hopefully help reduce these issues. Failing that, we might try more aggressive retries of dumpsys, or upgrade the emulator, or follow-up on comment 44.

Comment 51

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.