1607878 - Intermittent gv-junit runjunit.py | The previous test failed because of an error in the test harness | org.mozilla.geckoview.test.rule#TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms

Status: RESOLVED FIXED

(GeckoView :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aciure [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=284063816&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ccSYunWJRUKXT7KJzFnwRg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-01-08T18:55:53.345Z] 18:55:53 INFO - TEST-START | org.mozilla.geckoview.test.ExtensionActionTest.setBadgeBackgroundColor[#pageAction]
[task 2020-01-08T18:56:23.392Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: numtests=622
[task 2020-01-08T18:56:23.392Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stream=
[task 2020-01-08T18:56:23.392Z] 18:56:23 INFO - org.mozilla.geckoview.test | Error in setBadgeBackgroundColor#pageAction:
[task 2020-01-08T18:56:23.393Z] 18:56:23 INFO - org.mozilla.geckoview.test | org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.initTest(GeckoSessionTestRule.java:1285)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.lambda$evaluate$0$GeckoSessionTestRule$2(GeckoSessionTestRule.java:1297)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.-$$Lambda$GeckoSessionTestRule$2$sIbRNaZJgAu-QrUVWSGD8JbPSWM.run(lambda)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1950)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.os.Looper.loop(Looper.java:154)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.app.ActivityThread.main(ActivityThread.java:6077)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at java.lang.reflect.Method.invoke(Native Method)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | Caused by: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutRunnable.run(UiThreadUtils.java:58)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils.waitForCondition(UiThreadUtils.java:161)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:1142)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:1092)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.initTest(GeckoSessionTestRule.java:1281)
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | ... 10 more
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test |
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: test=setBadgeBackgroundColor[#pageAction]
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: class=org.mozilla.geckoview.test.ExtensionActionTest
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stack=org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - SUITE-END | took 73s
[task 2020-01-08T18:56:23.396Z] 18:56:23 WARNING - TEST-UNEXPECTED-FAIL | runjunit.py | The previous test failed because of an error in the test harness | org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-01-08T18:56:23.396Z] 18:56:23 INFO - Stopping web server
[task 2020-01-08T18:56:23.397Z] 18:56:23 INFO - Stopping web socket server
[task 2020-01-08T18:56:23.417Z] 18:56:23 INFO - Stopping ssltunnel
[task 2020-01-08T18:56:26.139Z] 18:56:26 ERROR - Return code: 255
[task 2020-01-08T18:56:26.139Z] 18:56:26 ERROR - No tests run or test summary not found
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - TinderboxPrint: geckoview-junit<br/><em class="testfail">T-FAIL</em>
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - ##### geckoview-junit log ends
[task 2020-01-08T18:56:26.140Z] 18:56:26 WARNING - # TBPL WARNING #
[task 2020-01-08T18:56:26.140Z] 18:56:26 WARNING - setting return code to 1
[task 2020-01-08T18:56:26.140Z] 18:56:26 WARNING - The geckoview-junit suite: geckoview-junit ran with return status: WARNING
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - Running post-action listener: _package_coverage_data
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - Running post-action listener: _resource_record_post_action
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - Running post-action listener: process_java_coverage_data
[task 2020-01-08T18:56:26.140Z] 18:56:26 INFO - Running post-action listener: stop_device

Comment 1

[Daniel Varga [:dvarga]]

Agi:, this seems to start after Bug 1581345 got landed. Can you please take a look at this?

Comment 2

[[ex-Mozilla] Agi Sferro | :agi]

That's on purpose. These test failures happen before the test even run, so the test code is not at fault, but the intermittent was billed against the test anyway. This causes a lot of one-off intermittents that are not actually intermittent. Now we have a new intermittent category for it.

Comment 3

[Intermittent Failures Robot]

26 failures in 4472 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 25
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to :Agi | ⏰ PST | he/him from comment #2)

That's on purpose. These test failures happen before the test even run, so the test code is not at fault, but the intermittent was billed against the test anyway. This causes a lot of one-off intermittents that are not actually intermittent. Now we have a new intermittent category for it.

Hmm, I don't think that's working correctly. The log here shows that we start org.mozilla.geckoview.test.ExtensionActionTest.setBadgeBackgroundColor[#pageAction], but never finish it. Clearly in this case the timeout should be charged to that test.

Comment 5

[[ex-Mozilla] Agi Sferro | :agi]

Yes that's expected, after junit thinks the test is starting we run a bunch of stuff (create a session, wait for initial about:blank, etc) all that stuff doesn't depend on the test.

You can see the code here: https://searchfox.org/mozilla-central/rev/c52d5f8025b5c9b2b4487159419ac9012762c40c/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1258-1287 anything in that method is going to execute as part of the test but it's not really the test's fault if we thrown an exception there.

Another hint that this is not the test's fault is that the exception stacktrace originates in initTest which runs before the test.

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

Ah, I see. Thanks!

Comment 7

[Intermittent Failures Robot]

35 failures in 3773 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #40 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 35

Platform breakdown:

• android-em-7-0-x86_64: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 8

[Intermittent Failures Robot]

17 failures in 838 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 2

Platform breakdown:

• android-em-7-0-x86_64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-01-23&endday=2020-01-23&tree=all

Comment 9

[Narcis Beleuzu [:NarcisB]]

There are 61 failures in the last 7 days. All of them in "android-em-7-0-x86_64 , debug".

Comment 10

[Intermittent Failures Robot]

54 failures in 4004 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 49
• try: 1
• mozilla-central: 4

Platform breakdown:

• android-em-7-0-x86_64: 54

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 2190 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 12

[Intermittent Failures Robot]

3 failures in 4045 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3

Platform breakdown:

• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 13

[Intermittent Failures Robot]

3 failures in 5233 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 1
• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-02-24&endday=2020-03-01&tree=all

Comment 14

[Intermittent Failures Robot]

2 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 15

[Intermittent Failures Robot]

15 failures in 1129 pushes (0.013 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 3
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-04-29&endday=2020-04-29&tree=all

Comment 16

[Cosmin Sabou [:CosminS]]

Looks like the spike that started from the 28th is caused by Bug 1618311. Range with retriggers.
Last 7 days view: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-04-24&endday=2020-05-01&tree=trunk&bug=1607878
Ed, could you please have a look over what's going on here?
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=300411286&repo=autoland

[task 2020-05-01T17:59:30.401Z] 17:59:30     INFO -  TEST-START | org.mozilla.geckoview.test.SessionLifecycleTest.collectOpen
[task 2020-05-01T18:00:00.349Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: numtests=716
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stream=
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | Error in collectOpen(org.mozilla.geckoview.test.SessionLifecycleTest):
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.initTest(GeckoSessionTestRule.java:1271)
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.lambda$evaluate$0$GeckoSessionTestRule$2(GeckoSessionTestRule.java:1283)
[task 2020-05-01T18:00:00.352Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.-$$Lambda$GeckoSessionTestRule$2$sIbRNaZJgAu-QrUVWSGD8JbPSWM.run(lambda)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1950)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.os.Looper.loop(Looper.java:154)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.app.ActivityThread.main(ActivityThread.java:6077)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at java.lang.reflect.Method.invoke(Native Method)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | Caused by: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutRunnable.run(UiThreadUtils.java:52)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.util.UiThreadUtils.waitForCondition(UiThreadUtils.java:155)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:1142)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:1090)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.initTest(GeckoSessionTestRule.java:1267)
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | 	... 10 more
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test |
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: test=collectOpen
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: class=org.mozilla.geckoview.test.SessionLifecycleTest
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stack=org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  SUITE-END | took 663s
[task 2020-05-01T18:00:00.353Z] 18:00:00  WARNING -  TEST-UNEXPECTED-FAIL | runjunit.py | The previous test failed because of an error in the test harness | org.mozilla.geckoview.test.rule.TestHarnessException: org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  Stopping web server
[task 2020-05-01T18:00:00.353Z] 18:00:00     INFO -  Stopping web socket server
[task 2020-05-01T18:00:00.373Z] 18:00:00     INFO -  Stopping ssltunnel
[task 2020-05-01T18:00:07.763Z] 18:00:07    ERROR - Return code: 255
[task 2020-05-01T18:00:07.763Z] 18:00:07    ERROR - No tests run or test summary not found
[task 2020-05-01T18:00:07.763Z] 18:00:07     INFO - TinderboxPrint: geckoview-junit<br/><em class="testfail">T-FAIL</em>
[task 2020-05-01T18:00:07.763Z] 18:00:07     INFO - ##### geckoview-junit log ends
[task 2020-05-01T18:00:07.763Z] 18:00:07  WARNING - # TBPL WARNING #
[task 2020-05-01T18:00:07.763Z] 18:00:07  WARNING - setting return code to 1

Comment 17

[Ed Lee :Mardak]

(In reply to Cosmin Sabou [:CosminS] from comment #16)

Looks like the spike that started from the 28th is caused by Bug 1618311. Range with retriggers.

Seems quite unlikely bug 1618311 regressed this. According to the range with retriggers, the previous commit is also failing the test frequently and one more previous is bug 1622060 that seems to have made gv-junit show up as gv-junit-e10s-multi and the test was also failing there too.

aklotz, would running the tests with dom.ipc.processCount=3 cause the intermittent timeouts to increase?

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=junit%2Copt&tochange=5690715ef70efb9e6c835e0e0ae63371af1475f7&fromchange=92172b9cef6b88934c7d54beb7e3d311a69df127

Comment 18

[(No longer employed by Mozilla) Aaron Klotz]

Yes, it would; this is not a regression, but rather a reflection of the fact that we introduced a new test suite with a new configuration.

Comment 19

[[ex-Mozilla] Agi Sferro | :agi]

I'm looking at this failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300412500&repo=autoland&lineNumber=11244 and it looks like our content process gets killed?

05-01 19:18:29.835  2806  2825 D GeckoViewProgress: observe: topic=oop-frameloader-crashed
05-01 19:18:29.835  2806  2825 D GeckoViewContent: observe: oop-frameloader-crashed
05-01 19:18:29.835  2806  2825 D GeckoViewProgress: observe: topic=oop-frameloader-crashed
05-01 19:18:29.835  2806  2825 D GeckoViewContent: observe: oop-frameloader-crashed
05-01 19:18:29.835  2806  2825 D GeckoViewProgress: observe: topic=oop-frameloader-crashed
05-01 19:18:29.836  2806  2825 D GeckoViewContent: observe: oop-frameloader-crashed
05-01 19:18:29.836  2806  2825 D GeckoViewProgress: observe: topic=oop-frameloader-crashed
05-01 19:18:29.836  2806  2825 D GeckoViewContent: observe: oop-frameloader-crashed

full log here: https://firefoxci.taskcluster-artifacts.net/B0WI9RMHSMS2FTu_FVPuCg/0/public/test_info//logcat-emulator-5554.log

I wonder if we can add some loginc in GeckoSessionTestRule to do something for this case.

Comment 20

[Intermittent Failures Robot]

21 failures in 691 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 4
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-01&endday=2020-05-01&tree=all

Comment 21

[Intermittent Failures Robot]

75 failures in 5423 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 7
• autoland: 56
• mozilla-central: 12

Platform breakdown:

• android-em-7-0-x86_64: 74
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 22

[Intermittent Failures Robot]

30 failures in 929 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 6
• try: 3
• mozilla-beta: 2

Platform breakdown:

• android-em-7-0-x86_64: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-04&endday=2020-05-04&tree=all

Comment 23

[Intermittent Failures Robot]

16 failures in 1176 pushes (0.014 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 4
• try: 2
• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-05&endday=2020-05-05&tree=all

Comment 24

[Intermittent Failures Robot]

25 failures in 931 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 5
• try: 1
• mozilla-beta: 2

Platform breakdown:

• android-em-7-0-x86_64: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-06&endday=2020-05-06&tree=all

Comment 25

[Intermittent Failures Robot]

27 failures in 980 pushes (0.028 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 17
• mozilla-central: 5
• mozilla-beta: 2
• try: 3

Platform breakdown:

• android-em-7-0-x86_64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-07&endday=2020-05-07&tree=all

Comment 26

[Intermittent Failures Robot]

18 failures in 869 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 3
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-08&endday=2020-05-08&tree=all

Comment 27

[Cosmin Sabou [:CosminS]]

Aaron, Agi, is there anything that can be done here to reduce the failure rate?
It has 121 total failures in the last week: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-05-02&endday=2020-05-09&tree=trunk&bug=1607878, 171 total failures since it was filed and it's on the disable-recommend bugs list.

Comment 28

[(No longer employed by Mozilla) Aaron Klotz]

A couple of issues here:

• This bug was filed for the gv-junit test suite. Now we've brought up the new gv-junit-e10s-multi test suite, where the same failures are occurring at a higher rate. I think this one should be kept for gv-junit and the gv-junit-e10s-multi failures should be tagged against a new, distinct bug from this one. I filed bug 1636720 for the latter.

• gv-junit-e10-multi is a new test suite. I'm working to get that cleaned up, but I feel like we're being too hasty in trying to disable tests for a brand new suite. I don't see how I am supposed to be able to stabilize it in CI if I'm not allowed to run it in CI. Running gv-junit-e10s-multi as tier-2 is also not a very good option for us, as we'd like anything that permaoranges the suite the be immediately backed out.

Comment 29

[(No longer employed by Mozilla) Aaron Klotz]

Removing this bug from the 78 sprint in favour of the e10s-multi variant.

Comment 30

[Intermittent Failures Robot]

124 failures in 5276 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 82
• mozilla-central: 25
• try: 10
• mozilla-beta: 7

Platform breakdown:

• android-em-7-0-x86_64: 124

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 31

[Cosmin Sabou [:CosminS]]

Failures here are actually bug 1636720.

Comment 32

[Intermittent Failures Robot]

10 failures in 5488 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 5
• mozilla-beta: 4
• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 33

[Intermittent Failures Robot]

3 failures in 4727 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3

Platform breakdown:

• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 34

[Intermittent Failures Robot]

4 failures in 5458 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-release: 2
• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 35

[Intermittent Failures Robot]

2 failures in 5211 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 36

[Intermittent Failures Robot]

3 failures in 4035 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 2

Platform breakdown:

• android-em-7-0-x86_64: 1
• android-em-7-0-x86_64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 37

[Intermittent Failures Robot]

5 failures in 5252 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 3

Platform breakdown:

• android-em-7-0-x86_64-shippable: 4
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607878&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 38

[(No longer employed by Mozilla) Aaron Klotz]

No more failures since bug 1647470 landed. I think we can call this resolved!

Comment 39

[[ex-Mozilla] Agi Sferro | :agi]

6c6f5da9-bf33-4082-a19d-7d99e8083d97 Mass rename intermittents due to Bug 1723034.