Closed Bug 1319196 Opened 3 years ago Closed 2 years ago

Timeout exceeded for shell call 'adb shell dumpsys window windows; echo $?'

Categories

(Firefox for Android :: Testing, defect, P3)

defect

Tracking

()

RESOLVED DUPLICATE of bug 1412297

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(3 files)

Our Android browser test harnesses (for mochitest, reftest, and robocop) use dumpsys to check that Fennec is in the foreground. Recently (last week, two weeks ago?) we started getting low-frequency intermittent test failures where the dumpsys call timed out, causing the test harness to abort the tests.

I'll try to dup related bugs here.

I have also been seeing these failures regularly in experimental mochitest-chrome runs on x86 - bug 1315749. Those failures appear to go away with the fix for bug 1318245, so I'm hoping this bug will dup to 1318245.


For x86 mochitest chrome compare:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76fc3f25d11b5586cd076f703c6565de1cdbbe95

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d13432af39a719f89edae42be69cc9aae9b897f
Duplicate of this bug: 1318457
Blocks: 1309088
Priority: -- → P3
Blocks: 1320354
Duplicate of this bug: 1320414
Duplicate of this bug: 1320415
Duplicate of this bug: 1320492
I notice "unable to restore focus" in many of the recent reports, just prior to the dumpsys timeout.
...typically following test_session_zombification.html in mochitest-chrome.
No longer blocks: 1320354
Duplicate of this bug: 1320354
Depends on: 1320987
Duplicate of this bug: 1320975
Duplicate of this bug: 1321629
Duplicate of this bug: 1320971
Duplicate of this bug: 1322157
Keywords: leave-open
test_session_zombification.html issue was likely addressed by bug 1320987, but other similar failures persist. I'm quite puzzled -- I can't find or think of any android harness or environment changes that might account for this new type of failure.
I wonder if increasing the timeout will help. If it doesn't help, I'll back this out.
Attachment #8818112 - Flags: review?(jmaher)
Comment on attachment 8818112 [details] [diff] [review]
increase dumpsys timeout

Review of attachment 8818112 [details] [diff] [review]:
-----------------------------------------------------------------

can you add a comment to indicate why you chose 60?
Attachment #8818112 - Flags: review?(jmaher) → review+
Sure: "Increased timeout to 60 seconds following intermittent timeouts at 30."
I'd also like to capture the screen when this happens, in case that shows something interesting. Performing the screenshot in kill() should also provide a capture on timeouts or other abnormal conditions resulting in the harness forcing the process down.
Attachment #8818314 - Flags: review?(jmaher)
Comment on attachment 8818314 [details] [diff] [review]
try to get a screenshot

Review of attachment 8818314 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!
Attachment #8818314 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c902a14dffee
Increase timeout for 'adb shell dumpsys'; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/f21a93061f9a
Take screenshot when dumpsys fails; r=jmaher
See Also: → 1178554
Duplicate of this bug: 1325528
Duplicate of this bug: 1325739
Duplicate of this bug: 1328478
Duplicate of this bug: 1328483
Screenshots are working great, but don't show anything out of the ordinary.

I don't think the timeout increase was effective.
Recent failures (today's dups, above) are all Android x86 mochitest-chrome; we just started running that job a few weeks ago.

In those logs, the anr report shows dumpsys running and blocking another window manager thread (if I'm reading that right).
Duplicate of this bug: 1326468
Duplicate of this bug: 1326469
Duplicate of this bug: 1326558
We can call dumpsys less often, by calling getTopActivity() less - only when the harness has not received log output recently. Call it less, hopefully it will fail less!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36de8eb4c9c962b0022e97136fd7c7855cce7bfe
Attachment #8824582 - Flags: review?(jmaher)
Comment on attachment 8824582 [details] [diff] [review]
call dumpsys less often

Review of attachment 8824582 [details] [diff] [review]:
-----------------------------------------------------------------

nice!
Attachment #8824582 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f02e4baff85d
Call dumpsys less during Android tests; r=jmaher
Duplicate of this bug: 1329244
Duplicate of this bug: 1329246
Duplicate of this bug: 1331642
It looks like comment 38 helped reduce the frequency of this problem quite a lot; since landing, I've only seen bug 1331642. Next step here might be ignoring dumpsys failures and retrying, cautiously.
Duplicate of this bug: 1305255
Bug 1335501 began triggering job retry for a very similar, but slightly different, error message. We can consider the same approach here, if needed.
I think this problem is still possible. I want to keep the bug open in case test failures return. However, I'm not going to work on this otherwise.
Assignee: gbrown → nobody
See Also: → 1364750
Assignee: nobody → gbrown
See Also: → 1402358
See Also: → 1412297
Bug 1412297 will hopefully help reduce these issues. Failing that, we might try more aggressive retries of dumpsys, or upgrade the emulator, or follow-up on comment 44.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1412297
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.