1506276 - Android 7.0 x86 geckoview-junit perma-fail: TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds after testScroll

Status: RESOLVED FIXED

(GeckoView :: General, defect, P3)

Description

[Geoff Brown [:gbrown]]

It looks like testScroll is hanging:

https://treeherder.mozilla.org/logviewer.html#?job_id=210879504&repo=mozilla-central&lineNumber=1537

[task 2018-11-09T18:48:13.736Z] 18:48:13     INFO -  TEST-START | org.mozilla.geckoview.test.AccessibilityTest.testScroll
[task 2018-11-09T19:27:59.484Z] 19:27:59  WARNING -  TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds


Note that the geckoview-junit tests on Android 7.0 x86 are running as tier 3 tasks currently (because they keep failing): That means they are hidden by default and not sheriffed. Failures are not starred by sheriffs, so they don't show up in the intermittent-failures reports. 


You can view the tasks on mozilla-central with:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=android%2C7.%2C0%2Cgeckoview-junit

Comment 1

[Geoff Brown [:gbrown]]

I think the hang started in this range:

https://treeherder.mozilla.org/#/jobs?repo=autoland&tier=1%2C2%2C3&tochange=8dd5d3eb6ea4a4883ec691eb83b82e34e56f7187&fromchange=f796b2c950e0689ad60163afca97f63e99f81f9d&searchStr=android%2C7.0%2Cjunit

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #1)
> I think the hang started in this range:

Yes. org.mozilla.geckoview.test.AccessibilityTest.testScroll started hanging with bug 1432019 (it had been failing cleanly earlier).

Comment 3

[Geoff Brown [:gbrown]]

I tried skipping (@Ignore) testScroll, but then the timeout moved to noPendingCallbacks_withSpecificSession...and if I skip testScroll and noPendingCallbacks_withSpecificSession, then noPendingCallbacks hangs.


:kats -- Can you have a look, since we were not seeing this before bug 1432019?

Note that geckoview-junit on Android 7.0 x86 runs as tier 3, so is hidden by default on treeherder. You can view the tasks on mozilla-central with:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=android%2C7.%2C0%2Cgeckoview-junit

You can run locally with 'mach geckoview-junit'. More info at https://developer.mozilla.org/en-US/docs/Mozilla/Geckoview-Junit_Tests.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

So bug 1432019 changes the thread on which we dispatch vsync events from some anonymous worker thread, to the main java UI thread. testScroll looks like it runs on the java UI thread, and does SystemClock.sleep calls between scrolls. So probably the vsync events are never getting dispatched because the UI thread is sleeping, and that causes everything to get jammed up. The test will need to be modified to relinquish the UI thread and post a runnable (or whatever the kotlin equivalent is) to do the rest of the work instead of sleeping and blocking the UI thread.

I'm not sure what the problem with noPendignCallbacks* is, it's not obvious to me what those tests are actually doing.

Comment 5

[Geoff Brown [:gbrown]]

Thanks kats.

:eeejay - Can you follow-up, at least for testScroll?

Comment 6

[Kartikaya Gupta (email:[e-mail])]

The noPendingCallback* tests look like they try to "open a session" which involves running the code at [1] on the main thread, which appears to be a blocking I/O call, maybe? So that will have the same problem in that will prevent vsync events from getting though and likely jam up stuff. The RDP connection code should probably be moved off the UI thread.

[1] https://searchfox.org/mozilla-central/rev/c0b26c40769a1e5607a1ae8be37fe64df64fc55e/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1234

Comment 7

[Eitan Isaacson [:eeejay]]

This test ultimately synthesizes a scroll wheel event in the chrome process. The scroll events it listens for are dispatched directly from layout via nsIScrollPositionListener[1].

I reverted the patch that kats mentions locally, no luck. I'll look at this further..

1. https://searchfox.org/mozilla-central/source/accessible/generic/DocAccessible.cpp#629

Comment 8

[Intermittent Failures Robot]

1 failures in 2387 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1506276&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 9

[Eitan Isaacson [:eeejay]]

Attachment: Add vanilla scroll test

Comment 10

[Eitan Isaacson [:eeejay]]

:kats

I did what you suggested (stopped blocking main thread), and I got only partial success. It seems like there are a lot of timing issues with APZC that are not deterministic enough for tests. Attached above is a non-a11y test to show those two issues.

There are two (non-UI thread blocking) sleeps there that make the test pass, but they seem kind of arbitrary and I don't think its a good way to write tests. The first sleep is after the page load so that the APZC has a chance to settle down before the first scrollIntoView so that it does not interfere. The second one is between the scrollIntoView and the mouse wheel simulation. From a bunch of logging it looks like we can get caught in the crossfire between different IPC APZC updates between content and chrome.

I guess my question for kats is: is there a way to write such a test without arbitrary waits.

Note: The accessibility scroll test also needs to sleep for 100ms between tests, but that is a specific amount of time that is needed so that the a11y layer can distinguish one scroll from the next (a11y scroll events are throttled to no more than on in 100ms).

Comment 11

[Kartikaya Gupta (email:[e-mail])]

We do have APIs that we use in APZ mochitests that do what you want without arbitrary waits (see e.g. [1] and [2] which would be most relevant to your current sleeps). However you'd need a bunch of plumbing to expose them properly to Java/Kotlin code. Is there any particular reason these tests need to be written on the Java side? Could you write them as mochitests instead?

[1] https://searchfox.org/mozilla-central/rev/d35199ef15ffa57d190886e20d5df6471faf0870/gfx/layers/apz/test/mochitest/apz_test_utils.js#312
[2] https://searchfox.org/mozilla-central/rev/d35199ef15ffa57d190886e20d5df6471faf0870/gfx/layers/apz/test/mochitest/apz_test_utils.js#127

Comment 12

[Eitan Isaacson [:eeejay]]

We already have mochitests for the scroll events that run on desktop. This is more of a platform test to see that it works right on Android.

We actually can wait for JS promises in kotlin, but I don't think we have apz_test_utils.js available to us..

I'm going to propose we disable this test everywhere with a reference to this bug if we ever decide to try to tackle this.

Comment 13

[Eitan Isaacson [:eeejay]]

Attachment: Disable AccessibilityTest#testScroll until we figure out APZC. r?yzen

Comment 14

[Yura Zenevich [:yzen] (please use "needinfo")]

Comment on attachment 9027725 [details] [diff] [review]
Disable AccessibilityTest#testScroll until we figure out APZC. r?yzen

Review of attachment 9027725 [details] [diff] [review]:
-----------------------------------------------------------------

sounds good

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/31b4c8ac21fd
Disable AccessibilityTest#testScroll until we figure out APZC. r=yzen

Comment 16

[Geoff Brown [:gbrown]]

We expect the other tests to hang still, right? Is there a plan for those? 

Should we leave this bug open or file a new one?

Comment 17

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/31b4c8ac21fd

Comment 18

[Chris Peterson [:cpeterson]]

64=wontfix because we don't need to uplift this test fix to Beta.

Comment 19

[Chris Peterson [:cpeterson]]

status-geckoview64=wontfix

Comment 20

[Intermittent Failures Robot]

2 failures in 2808 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 2

Platform breakdown:
* android-em-7-0-x86: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1506276&startday=2018-11-26&endday=2018-12-02&tree=all