Open Bug 1554779 Opened 5 years ago Updated 1 day ago

Intermittent TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 3000 seconds

Categories

(GeckoView :: General, defect, P3)

Unspecified
Android

Tracking

(firefox69 wontfix, firefox72 fix-optional)

REOPENED
Tracking Status
firefox69 --- wontfix
firefox72 --- fix-optional

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure, leave-open, regression, Whiteboard: [geckoview:m1912][geckoview:m74][retriggered][stockwell unknown])

Attachments

(2 files)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=248574001&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/c_rcFzLmQ6SvsD9dOfG3lA/runs/0/artifacts/public/logs/live_backing.log


task 2019-05-27T16:47:05.671Z] 16:47:05 INFO - TEST-START | org.mozilla.geckoview.test.WebExtensionTest.iframeTopLevel
[task 2019-05-27T16:47:24.313Z] 16:47:24 WARNING - TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds
[task 2019-05-27T16:47:24.313Z] 16:47:24 INFO - Passed: 144
[task 2019-05-27T16:47:24.313Z] 16:47:24 INFO - Failed: 0
[task 2019-05-27T16:47:24.313Z] 16:47:24 INFO - Todo: 14
[task 2019-05-27T16:47:24.313Z] 16:47:24 INFO - SUITE-END | took 2400s
[task 2019-05-27T16:47:25.739Z] 16:47:25 INFO - Stopping web server
[task 2019-05-27T16:47:25.748Z] 16:47:25 INFO - Stopping web socket server
[task 2019-05-27T16:47:25.769Z] 16:47:25 INFO - Stopping ssltunnel
[task 2019-05-27T16:47:29.026Z] 16:47:29 INFO - Return code: 0

OS: All → Android

[bulk change --> 69 fix-optional b/c intermittent]

There are 27 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-01&endday=2019-06-08&tree=trunk&bug=1554779

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250752231&repo=autoland&lineNumber=2254

[task 2019-06-08T10:13:10.571Z] 10:13:10 INFO - TEST-START | org.mozilla.geckoview.test.WebExtensionTest.backgroundPortDisconnect
[task 2019-06-08T10:14:16.220Z] 10:14:16 WARNING - TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds
[task 2019-06-08T10:14:16.220Z] 10:14:16 INFO - Passed: 42
[task 2019-06-08T10:14:16.220Z] 10:14:16 INFO - Failed: 0
[task 2019-06-08T10:14:16.220Z] 10:14:16 INFO - Todo: 6
[task 2019-06-08T10:14:16.220Z] 10:14:16 INFO - SUITE-END | took 2400s
[task 2019-06-08T10:14:17.638Z] 10:14:17 INFO - Stopping web server
[task 2019-06-08T10:14:17.648Z] 10:14:17 INFO - Stopping web socket server
[task 2019-06-08T10:14:17.669Z] 10:14:17 INFO - Stopping ssltunnel
[task 2019-06-08T10:14:21.226Z] 10:14:21 INFO - Return code: 0
[task 2019-06-08T10:14:21.226Z] 10:14:21 INFO - TinderboxPrint: geckoview-junit<br/>42/0
[task 2019-06-08T10:14:21.226Z] 10:14:21 INFO - ##### geckoview-junit log ends
[task 2019-06-08T10:14:21.226Z] 10:14:21 WARNING - # TBPL WARNING #
[task 2019-06-08T10:14:21.226Z] 10:14:21 WARNING - setting return code to 1
[task 2019-06-08T10:14:21.226Z] 10:14:21 WARNING - The geckoview-junit suite: geckoview-junit ran with return status: WARNING
[task 2019-06-08T10:14:21.227Z] 10:14:21 INFO - Running post-action listener: _package_coverage_data
[task 2019-06-08T10:14:21.227Z] 10:14:21 INFO - Running post-action listener: _resource_record_post_action
[task 2019-06-08T10:14:21.227Z] 10:14:21 INFO - Running post-action listener: process_java_coverage_data
[task 2019-06-08T10:14:21.227Z] 10:14:21 INFO - Running post-action listener: stop_device
[task 2019-06-08T10:14:22.599Z] 10:14:22 INFO - Killing logcat pid 917.
[task 2019-06-08T10:14:22.600Z] 10:14:22 INFO - Killing every process called emulator64-arm
[task 2019-06-08T10:14:22.612Z] 10:14:22 INFO - Killing pid 486.
[task 2019-06-08T10:14:22.612Z] 10:14:22 INFO - [mozharness: 2019-06-08 10:14:22.612217Z] Finished run-tests step (success)
[task 2019-06-08T10:14:22.612Z] 10:14:22 INFO - Running post-run listener: _resource_record_post_run
[task 2019-06-08T10:14:22.781Z] 10:14:22 INFO - Total resource usage - Wall time: 2481s; CPU: 25.0%; Read bytes: 14622720; Write bytes: 1484517376; Read time: 384; Write time: 254828
[task 2019-06-08T10:14:22.781Z] 10:14:22 INFO - TinderboxPrint: CPU usage<br/>25.2%
[task 2019-06-08T10:14:22.781Z] 10:14:22 INFO - TinderboxPrint: I/O read bytes / time<br/>14,622,720 / 384
[task 2019-06-08T10:14:22.781Z] 10:14:22 INFO - TinderboxPrint: I/O write bytes / time<br/>1,484,517,376 / 254,828
[task 2019-06-08T10:14:22.782Z] 10:14:22 INFO - TinderboxPrint: CPU idle<br/>7,304.3 (74.5%)
[task 2019-06-08T10:14:22.782Z] 10:14:22 INFO - TinderboxPrint: CPU user<br/>2,470.1 (25.2%)
[task 2019-06-08T10:14:22.782Z] 10:14:22 INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2019-06-08T10:14:22.783Z] 10:14:22 INFO - verify-device - Wall time: 6s; CPU: 9.0%; Read bytes: 8192; Write bytes: 736616448; Read time: 0; Write time: 139048
[task 2019-06-08T10:14:22.785Z] 10:14:22 INFO - install - Wall time: 62s; CPU: 25.0%; Read bytes: 0; Write bytes: 384094208; Read time: 0; Write time: 71856
[task 2019-06-08T10:14:22.810Z] 10:14:22 INFO - run-tests - Wall time: 2412s; CPU: 25.0%; Read bytes: 12558336; Write bytes: 363462656; Read time: 368; Write time: 43916
[task 2019-06-08T10:14:23.576Z] 10:14:23 WARNING - returning nonzero exit status 1
[task 2019-06-08T10:14:23.625Z] cleanup
[task 2019-06-08T10:14:23.625Z] + cleanup
[task 2019-06-08T10:14:23.626Z] + local rv=1
[task 2019-06-08T10:14:23.626Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2019-06-08T10:14:23.626Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2019-06-08T10:14:23.630Z] + true
[task 2019-06-08T10:14:23.630Z] + cleanup_xvfb
[task 2019-06-08T10:14:23.630Z] pidof Xvfb
[task 2019-06-08T10:14:23.631Z] ++ pidof Xvfb
[task 2019-06-08T10:14:23.635Z] + local xvfb_pid=26
[task 2019-06-08T10:14:23.635Z] + local vnc=false
[task 2019-06-08T10:14:23.635Z] + local interactive=false
[task 2019-06-08T10:14:23.644Z] + '[' -n 26 ']'
[task 2019-06-08T10:14:23.644Z] + [[ false == false ]]
[task 2019-06-08T10:14:23.644Z] + [[ false == false ]]
[task 2019-06-08T10:14:23.645Z] + kill 26
[task 2019-06-08T10:14:23.645Z] + screen -XS xvfb quit
[task 2019-06-08T10:14:23.645Z] XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
[task 2019-06-08T10:14:23.645Z] after 1066 requests (1066 known processed) with 1 events remaining.
[task 2019-06-08T10:14:23.645Z] compizconfig - Info: Backend : ini
[task 2019-06-08T10:14:23.645Z] compizconfig - Info: Integration : true
[task 2019-06-08T10:14:23.645Z] compizconfig - Info: Profile : default
[task 2019-06-08T10:14:23.846Z] No screen session found.
[task 2019-06-08T10:14:23.846Z] + true
[task 2019-06-08T10:14:23.846Z] + exit 1
[taskcluster 2019-06-08 10:14:24.169Z] === Task Finished ===
[taskcluster 2019-06-08 10:14:29.113Z] Unsuccessful task run with exit code: 1 completed in 2772.014 seconds

Whiteboard: [stockwell needswork:owner]

Chris, can you please take a look at this?
It's failing quite frequently along with Bug 1592257.

Flags: needinfo?(cpeterson)

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

Sending to GeckoView triage for prioritization of this frequent intermittent test failure.

Flags: needinfo?(cpeterson)
Priority: P5 → --
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

The test logs here look a little random: They seem to hang at around the same place, but there is quite a bit of variation. I think the hang may be interfering with the test log.

Logcat is likely a better indicator. Again, I see variation across failure instances, but the majority seem to hang after SessionLifecycleTest.collectOpen. This seems typical:

https://firefoxci.taskcluster-artifacts.net/U3UNIqCzQJSawxqA23tXPg/0/public/test_info//logcat-emulator-5554.log

11-26 20:13:56.115  2786  2811 D GeckoViewXUL: receiveMessage GeckoView:ContentModuleLoaded {"module":"GeckoViewSettings"}
11-26 20:13:56.115  2786  2811 D GeckoViewModule: enableQueuing false
11-26 20:13:56.115  2786  2811 D GeckoViewModule: dispatchQueued
11-26 20:13:56.115  2786  2811 D GeckoViewXUL: receiveMessage GeckoView:ContentModuleLoaded {"module":"SessionStateAggregator"}
11-26 20:13:56.115  2786  2811 D GeckoViewXUL: receiveMessage GeckoView:ContentModuleLoaded {"module":"GeckoViewAutofill"}
11-26 20:13:56.115  3829  3844 D GeckoViewSettings[C]: onSettingsUpdate {"useMultiprocess":true,"chromeUri":null,"screenId":0,"userAgentOverride":null,"allowJavascript":true,"userAgentMode":0,"viewportMode":0,"useTrackingProtection":false,"suspendMediaWhenInactive":false,"usePrivateMode":false,"unsafeSessionContextId":null,"displayMode":0,"sessionContextId":null,"fullAccessibilityTree":false,"isPopup":false}
11-26 20:13:56.115  2786  2786 I art     : Explicit concurrent mark sweep GC freed 5(152B) AllocSpace objects, 0(0B) LOS objects, 30% free, 9MB/13MB, paused 66us total 7.217ms
11-26 20:13:56.115  2786  2786 I art     : Starting a blocking GC Explicit
11-26 20:13:56.122  2786  2786 I art     : Explicit concurrent mark sweep GC freed 4739(185KB) AllocSpace objects, 0(0B) LOS objects, 31% free, 8MB/12MB, paused 68us total 6.764ms
11-26 20:13:56.122  2786  2786 I art     : Starting a blocking GC Explicit
11-26 20:13:56.129  2786  2786 I art     : Explicit concurrent mark sweep GC freed 355(13KB) AllocSpace objects, 0(0B) LOS objects, 31% free, 8MB/12MB, paused 66us total 6.528ms
11-26 20:13:56.129  2786  2786 E GeckoSessionTestRule: after evaluate
11-26 20:13:56.129  2786  2786 E GeckoSessionTestRule: after performTestEndCheck
11-26 20:13:56.129  2786  2786 E GeckoSessionTestRule: ====
11-26 20:13:56.130  2786  2811 D GeckoViewProcessHangMonitor: onDestroy
11-26 20:13:56.131  3829  3844 D GeckoViewAutofill[C]: handleEvent: pagehide
11-26 20:13:56.131  3829  3844 D GeckoViewAutofill: Clearing auto-fill
11-26 20:13:56.190  2786  2801 I TestRunner: finished: collectOpen(org.mozilla.geckoview.test.SessionLifecycleTest)
11-26 20:13:56.191  2786  2811 D GeckoViewModule: unregisterListener
11-26 20:13:56.191  2786  2811 D GeckoViewNavigation: onDisable
11-26 20:13:56.191  2786  2811 D GeckoViewProcessHangMonitor: onDisable
11-26 20:13:56.191  2786  2811 D GeckoViewModule: unregisterListener
11-26 20:13:56.192  2786  2811 D GeckoViewProcessHangMonitor: onDestroy
11-26 20:13:56.192  2786  2811 D GeckoViewProgress: onDisable
11-26 20:13:56.192  2786  2811 D GeckoViewModule: unregisterListener
11-26 20:13:56.192  2786  2811 D GeckoViewModule: unregisterListener
11-26 20:13:56.198  3829  3844 D GeckoViewSelectionAction[C]: handleEvent: visibilitychange
11-26 20:13:56.198  3829  3844 D GeckoViewContent[C]: onDisable
11-26 20:13:56.198  3829  3844 D GeckoViewMedia[C]: onDisable
11-26 20:13:56.198  3829  3844 D GeckoViewScroll[C]: onDisable
11-26 20:13:56.198  3829  3844 D GeckoViewSelectionAction[C]: onDisable
11-26 20:13:56.199  3829  3844 D GeckoViewAutofill[C]: handleEvent: pagehide
11-26 20:13:56.199  3829  3844 D GeckoViewAutofill: Clearing auto-fill
...<no further geckoview logging>

:snorp -- Any idea?

Flags: needinfo?(snorp)

Joel, can you help here?

There are 280 total failures in the last 7 days and 484 total failures in the last 30 days.
Occurrences here started spiking since the 21st of November: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-11-04&endday=2019-12-04&tree=trunk&bug=1554779

Flags: needinfo?(jmaher)

:gbrown, can you disable these tests for now as we are not getting traction to fix them.

Flags: needinfo?(jmaher) → needinfo?(gbrown)
Whiteboard: [retriggered][stockwell disable-recommended] → [geckoview:m1912][retriggered][stockwell disable-recommended]
Flags: needinfo?(snorp)
Priority: -- → P1

I can avoid the 2400 second timeouts by skipping SessionLifecycleTest.collectOpen (comment 27) and readFromParcel_chained, but the suite continues to fail with about the same frequency:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2bc235f430eb2aa5966e4570abf0504da26cd58a

I'll continue looking into greening-by-skipping.

Flags: needinfo?(gbrown)

(In reply to Geoff Brown [:gbrown] from comment #39)

I'll continue looking into greening-by-skipping.

Still struggling...even with 14 tests skipped, I see random failures:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=279908030&revision=82ea0fe29b4b9f0f70b3c5f2e4d82aeca30f3cd2

The hang seems to be (at least partially) caused by Bug 1586986.

Hiro can you take a look at the above comment?

Flags: needinfo?(hikezoe.birchill)

OK, I will take a look. Agi, can you please provide any clues you've already found?

Assignee: nobody → hikezoe.birchill
Status: NEW → ASSIGNED
Flags: needinfo?(hikezoe.birchill) → needinfo?(agi)

I found a clue why Agi thinks bug 1586986 caused the hang.

From a failure log

[task 2019-11-25T11:40:34.136Z] 11:40:34 INFO - Thread 0 (crashed)
[task 2019-11-25T11:40:34.136Z] 11:40:34 INFO - 0 libxul.so!unsigned long nsTArray_Impl<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry*, nsTArrayInfallibleAllocator>::BinaryIndexOf<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> >(mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const* const&, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> const&) const [nsTArray.h:086f87f26b1fd6049e616399eb7e519b7b775671 : 1274 + 0x0]
-------------- snip --------------
[task 2019-11-25T11:40:34.142Z] 11:40:34 INFO - 7 libxul.so!nsWindow::LayerViewSupport::SetFixedBottomOffset(int)::{lambda()#1}::operator()() const [nsWindow.cpp:086f87f26b1fd6049e616399eb7e519b7b775671 : 1101 + 0x8]

SetFixedBottomOffset is actually called in the test I added in bug 1586986. So in that sense the test added in bug 1586986 introduced this hang, but I am pretty sure this hang (deadlock) has been there before bug 1586986. Before bug 1586986 we had no automated tests which call SetFixedBottomOffset and I actually had been seeing the deadlock locally without the changes for bug 1586986 (I asked it in a slack channel).

This is another deadlock in a failure before bug 1586986 landed.

So, there have been reasons causing the deadlock for a while, and bug 1586986 added a test which made the deadlock more frequent, and as per the orange factor logs, there seems more tests added which made it more and more frequent.

The problematic call of SetFixedBottomOffset is from GeckoSession.onCompositorReady, not from GeckoSession.setFixedBottomOffset.

It seems that the deadlock is solved by initializing mFixedBottomOffset with zero and not calling SetFixedBottomOffset in GeckoSession.onCompositorReady if it's not been changed since the initialization.

Here is a try;
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ef7834db5d565e7a722892baef8808775fa77e1

There are still lots of oranges, but no timed out happened.

And don't call setFixedBottomOffset in GeckoSession.onCompositorReady if it's not
been changed since the initialization.

Thank you so much hiro!

Flags: needinfo?(agi)
Pushed by apavel@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/29699e0b47b0
Initialize GeckoSession.mFixedBottomOffset with zero. r=geckoview-reviewers,snorp

Note that another deadlock happened in nsWindow::LayerViewSupport::OnDetach.

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=280379978&repo=autoland&lineNumber=10322

We may need to check mAttachedCompositor in onCompositorAttached and on CompositorDetached, I am totally unsure though. (I haven't seen this deadlock locally)

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

Hiroyuki I suggest to leave this open until we're sure it no longer appears, what do you think?

Flags: needinfo?(hikezoe.birchill)

Yep, I agree. Unassigning myself for now.

Assignee: hikezoe.birchill → nobody
Status: RESOLVED → REOPENED
Flags: needinfo?(hikezoe.birchill)
Resolution: FIXED → ---
Keywords: leave-open
Whiteboard: [geckoview:m1912][retriggered][stockwell disable-recommended] → [geckoview:m1912][retriggered][stockwell fixed:patch]
Target Milestone: mozilla73 → ---

This has has not completely gone away, however failure rate decreased significantly

Since the fix landed, there are 8 failures:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-10&endday=2019-12-12&tree=trunk&bug=1554779

(In reply to Andreea Pavel [:apavel] from comment #57)

This has has not completely gone away, however failure rate decreased significantly

Since the fix landed, there are 8 failures:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-10&endday=2019-12-12&tree=trunk&bug=1554779

I believe all failures are the deadlock in nsWindow::LayerViewSupport::OnDetach as I commented 52.

(In reply to Hiroyuki Ikezoe (:hiro) from comment #58)

(In reply to Andreea Pavel [:apavel] from comment #57)

This has has not completely gone away, however failure rate decreased significantly

Since the fix landed, there are 8 failures:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-10&endday=2019-12-12&tree=trunk&bug=1554779

I believe all failures are the deadlock in nsWindow::LayerViewSupport::OnDetach as I commented 52.

Hiroyuki do you know who can take a look at that?

Flags: needinfo?(hikezoe.birchill)

Someone in mobile team I think.

FWIW, I just pushed a try with mAttachedCompositor checks;
https://treeherder.mozilla.org/#/jobs?repo=try&revision=108099a3b77666225d55b75879c06beef27689b5

Flags: needinfo?(hikezoe.birchill) → needinfo?(snorp)

This patch is a little worrying to me, since we shouldn't be attaching if we're already attached and vice-versa.

Flags: needinfo?(snorp)

Emily, can you assign someone to take a look at this?

There are 20 total failures in the last 7 days on android-em-7-0-x86_64 debug.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=281487604&repo=autoland&lineNumber=5660

[task 2019-12-17T04:14:51.607Z] 04:14:51 INFO - TEST-FAIL | org.mozilla.geckoview.test.MediaElementTest.mp4FullscreenMedia | took 202ms
[task 2019-12-17T04:14:51.607Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: numtests=599
[task 2019-12-17T04:14:51.607Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stream=
[task 2019-12-17T04:14:51.607Z] 04:14:51 INFO - org.mozilla.geckoview.test | org.mozilla.geckoview.test.NavigationDelegateTest:
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: test=loadUriHeaderBadOverrides
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: class=org.mozilla.geckoview.test.NavigationDelegateTest
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: current=242
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS_CODE: 1
[task 2019-12-17T04:14:51.608Z] 04:14:51 INFO - TEST-START | org.mozilla.geckoview.test.NavigationDelegateTest.loadUriHeaderBadOverrides
[task 2019-12-17T04:52:10.040Z] 04:52:10 WARNING - TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds
[task 2019-12-17T04:52:10.040Z] 04:52:10 INFO - Passed: 217
[task 2019-12-17T04:52:10.040Z] 04:52:10 INFO - Failed: 0
[task 2019-12-17T04:52:10.041Z] 04:52:10 INFO - Todo: 24
[task 2019-12-17T04:52:10.041Z] 04:52:10 INFO - SUITE-END | took 2401s
[task 2019-12-17T04:52:10.665Z] 04:52:10 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpEJ5w7R/7f801193-ba0d-91e1-1cd5-a905072c4c0c.dmp /builds/worker/workspace/build/symbols
[task 2019-12-17T04:52:14.888Z] 04:52:14 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/7f801193-ba0d-91e1-1cd5-a905072c4c0c.dmp
[task 2019-12-17T04:52:14.894Z] 04:52:14 WARNING - PROCESS-CRASH | org.mozilla.geckoview.test.NavigationDelegateTest.loadUriHeaderBadOverrides | application crashed [@ unsigned long nsTArray_Impl<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry*, nsTArrayInfallibleAllocator>::BinaryIndexOf<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> >(mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const* const&, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> const&) const]
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Crash dump filename: /tmp/tmpEJ5w7R/7f801193-ba0d-91e1-1cd5-a905072c4c0c.dmp
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Operating system: Android
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - CPU: amd64
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - family 6 model 6 stepping 3
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - 4 CPUs
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - GPU: UNKNOWN
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Crash address: 0x0
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Process uptime: not available
[task 2019-12-17T04:52:14.895Z] 04:52:14 INFO - Thread 0 (crashed)
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - 0 libxul.so!unsigned long nsTArray_Impl<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry*, nsTArrayInfallibleAllocator>::BinaryIndexOf<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> >(mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const* const&, nsDefaultComparator<mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*> const&) const [nsTArray.h:54c2441b54abdbcc73e8c206da154c9b902550cd : 1274 + 0x0]
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - rax = 0x00007fffe432d4c0 rdx = 0x00007635a66c1bd8
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - rcx = 0xaaaaaaaaaaaaaaaa rbx = 0x00007635913b8300
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - rsi = 0x00007fffe432d4e8 rdi = 0x0000000000000000
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - rbp = 0x00007fffe432d4d0 rsp = 0x00007fffe432d4a0
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - r8 = 0x00007fffe432d4b8 r9 = 0x00007635a66c0cd0
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - r10 = 0x00007fffe432d498 r11 = 0x0000000000000000
[task 2019-12-17T04:52:14.896Z] 04:52:14 INFO - r12 = 0x00007635a6c043a0 r13 = 0x00007635928b98b8
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - r14 = 0x00007635a6c432f0 r15 = 0x0000000000000000
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - rip = 0x000076359ddc4619
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - Found by: given as instruction pointer in context
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - 1 libxul.so!mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::InTransitiveClosure(mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*, mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::OrderingEntry const*) const [DeadlockDetector.h:54c2441b54abdbcc73e8c206da154c9b902550cd : 271 + 0xf]
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - rbp = 0x00007fffe432d520 rsp = 0x00007fffe432d4e0
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - rip = 0x000076359ddc44a0
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - 2 libxul.so!mozilla::DeadlockDetector<mozilla::BlockingResourceBase>::CheckAcquisition(mozilla::BlockingResourceBase const*, mozilla::BlockingResourceBase const*) [DeadlockDetector.h:54c2441b54abdbcc73e8c206da154c9b902550cd : 241 + 0xe]
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - rbp = 0x00007fffe432d580 rsp = 0x00007fffe432d530
[task 2019-12-17T04:52:14.897Z] 04:52:14 INFO - rip = 0x000076359ddb7ea3
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - 3 libxul.so!mozilla::BlockingResourceBase::CheckAcquire() [BlockingResourceBase.cpp:54c2441b54abdbcc73e8c206da154c9b902550cd : 225 + 0x12]
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - rbp = 0x00007fffe432d630 rsp = 0x00007fffe432d590
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - rip = 0x000076359ddb7b36
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - 4 libxul.so!mozilla::OffTheBooksMutex::Lock() [BlockingResourceBase.cpp:54c2441b54abdbcc73e8c206da154c9b902550cd : 318 + 0xc]
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - rbp = 0x00007fffe432d650 rsp = 0x00007fffe432d640
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - rip = 0x000076359ddb8202
[task 2019-12-17T04:52:14.898Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - 5 libxul.so!nsWindow::WindowPtr<nsWindow::LayerViewSupport>::Locked::Locked(nsWindow::WindowPtr<nsWindow::LayerViewSupport>&) [nsWindow.h:54c2441b54abdbcc73e8c206da154c9b902550cd : 126 + 0xc]
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - rbp = 0x00007fffe432d690 rsp = 0x00007fffe432d660
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - rip = 0x00007635a04ae878
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - 6 libxul.so!nsWindow::LayerViewSupport::OnDetach(already_AddRefed<mozilla::Runnable>)::{lambda()#1}::operator()() const [nsWindow.cpp:54c2441b54abdbcc73e8c206da154c9b902550cd : 874 + 0x1b]
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - rbp = 0x00007fffe432d730 rsp = 0x00007fffe432d6a0
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - rip = 0x00007635a04b0330
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - 7 libxul.so!mozilla::detail::RunnableFunction<nsWindow::LayerViewSupport::OnDetach(already_AddRefed<mozilla::Runnable>)::{lambda()#1}>::Run() [nsThreadUtils.h:54c2441b54abdbcc73e8c206da154c9b902550cd : 564 + 0x9]
[task 2019-12-17T04:52:14.899Z] 04:52:14 INFO - rbp = 0x00007fffe432d740 rsp = 0x00007fffe432d740
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - rip = 0x00007635a04b029b
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - 8 libxul.so!mozilla::RunAndroidUiTasks() [AndroidUiThread.cpp:54c2441b54abdbcc73e8c206da154c9b902550cd : 331 + 0x11]
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - rbp = 0x00007fffe432d7b0 rsp = 0x00007fffe432d750
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - rip = 0x00007635a047ac57
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - Found by: previous frame's frame pointer
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - 9 dalvik-LinearAlloc (deleted) + 0x30a0
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - rbp = 0x00007fffe432d7b0 rsp = 0x00007fffe432d790
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - rip = 0x00007635bf6160a0
[task 2019-12-17T04:52:14.900Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - 10 dalvik-main space (deleted) + 0x28f00
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rbp = 0x00007fffe432d7b0 rsp = 0x00007fffe432d798
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rip = 0x0000000012c28f00
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - 11 dalvik-main space (deleted) + 0x2e288
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rbp = 0x00007fffe432d7b0 rsp = 0x00007fffe432d7a0
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rip = 0x0000000012c2e288
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - 12 dalvik-LinearAlloc (deleted) + 0x2df78
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rbp = 0x00007fffe432d7b0 rsp = 0x00007fffe432d7a8
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - rip = 0x00007635bf640f78
[task 2019-12-17T04:52:14.901Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - 13 dalvik-main space (deleted) + 0x2e1e8
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - rsp = 0x00007fffe432d7b8 rip = 0x0000000012c2e1e8
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - 14 base.odex + 0x927146
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - rsp = 0x00007fffe432d7c0 rip = 0x00007635a8a8e146
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - 15 dalvik-LinearAlloc (deleted) + 0x6aa50
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - rsp = 0x00007fffe432d7e8 rip = 0x00007635bf67da50
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - 16 dalvik-main space (deleted) + 0x2e288
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - rsp = 0x00007fffe432d808 rip = 0x0000000012c2e288
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - 17 dalvik-LinearAlloc (deleted) + 0x30a0
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - rsp = 0x00007fffe432d830 rip = 0x00007635bf6160a0
[task 2019-12-17T04:52:14.902Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - 18 dalvik-main space (deleted) + 0x2e1e8
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - rsp = 0x00007fffe432d838 rip = 0x0000000012c2e1e8
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - 19 dalvik-main space (deleted) + 0x28f00
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - rsp = 0x00007fffe432d840 rip = 0x0000000012c28f00
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - 20 dalvik-main space (deleted) + 0x2e288
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - rsp = 0x00007fffe432d848 rip = 0x0000000012c2e288
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - 21 dalvik-LinearAlloc (deleted) + 0x2df78
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - rsp = 0x00007fffe432d850 rip = 0x00007635bf640f78
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - 22 dalvik-jit-code-cache (deleted) + 0xf6d6
[task 2019-12-17T04:52:14.903Z] 04:52:14 INFO - rsp = 0x00007fffe432d860 rip = 0x00007635aabb56d6
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - 23 dalvik-LinearAlloc (deleted) + 0x30a0
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - rsp = 0x00007fffe432d868 rip = 0x00007635bf6160a0
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - 24 dalvik-main space (deleted) + 0x69bc00
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - rsp = 0x00007fffe432d878 rip = 0x000000001329bc00
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - 25 boot.oat + 0x65b6ee
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - rsp = 0x00007fffe432d880 rip = 0x0000000071fa26ee
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - 26 dalvik-main space (deleted) + 0x44c00
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - rsp = 0x00007fffe432d890 rip = 0x0000000012c44c00
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - 27 dalvik-main space (deleted) + 0x2e1e8
[task 2019-12-17T04:52:14.904Z] 04:52:14 INFO - rsp = 0x00007fffe432d898 rip = 0x0000000012c2e1e8
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - 28 dalvik-main space (deleted) + 0x28f00
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - rsp = 0x00007fffe432d8a0 rip = 0x0000000012c28f00
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - 29 dalvik-main space (deleted) + 0x28f00
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - rsp = 0x00007fffe432d8a8 rip = 0x0000000012c28f00
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - 30 boot-framework.oat + 0x1657220
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - rsp = 0x00007fffe432d8b0 rip = 0x00000000745aa220
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - 31 system@framework@boot-framework.art + 0x256538
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - rsp = 0x00007fffe432d8b8 rip = 0x00000000714f6538
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.905Z] 04:52:14 INFO - 32 libart.so + 0x138e43
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - rsp = 0x00007fffe432d8c0 rip = 0x00007635c0a44e43
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - 33 system@framework@boot.art + 0x50f2c8
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - rsp = 0x00007fffe432d8c8 rip = 0x0000000070fbe2c8
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - 34 dalvik-main space (deleted) + 0x2e1e8
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - rsp = 0x00007fffe432d8d0 rip = 0x0000000012c2e1e8
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - 35 dalvik-main space (deleted) + 0x7ee5c0
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - rsp = 0x00007fffe432d8d8 rip = 0x00000000133ee5c0
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - 36 boot-framework.oat + 0x1657296
[task 2019-12-17T04:52:14.906Z] 04:52:14 INFO - rsp = 0x00007fffe432d8e0 rip = 0x00000000745aa296
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - 37 system@framework@boot-framework.art + 0x256570
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - rsp = 0x00007fffe432d8e8 rip = 0x00000000714f6570
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - 38 dalvik-main space (deleted) + 0x44c00
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - rsp = 0x00007fffe432d8f0 rip = 0x0000000012c44c00
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - 39 dalvik-main space (deleted) + 0x69bc00
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - rsp = 0x00007fffe432d8f8 rip = 0x000000001329bc00
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - 40 dalvik-main space (deleted) + 0x95d8
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - rsp = 0x00007fffe432d900 rip = 0x0000000012c095d8
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.907Z] 04:52:14 INFO - 41 dalvik-main space (deleted) + 0x44c00
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - rsp = 0x00007fffe432d908 rip = 0x0000000012c44c00
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - 42 dalvik-main space (deleted) + 0x69bc00
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - rsp = 0x00007fffe432d910 rip = 0x000000001329bc00
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - 43 dalvik-main space (deleted) + 0x95d8
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - rsp = 0x00007fffe432d918 rip = 0x0000000012c095d8
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - 44 dalvik-jit-code-cache (deleted) + 0xa0d4
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - rsp = 0x00007fffe432d920 rip = 0x00007635aabb00d4
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - 45 dalvik-LinearAlloc (deleted) + 0x2df78
[task 2019-12-17T04:52:14.908Z] 04:52:14 INFO - rsp = 0x00007fffe432d928 rip = 0x00007635bf640f78
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - 46 dalvik-main space (deleted) + 0x44790
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - rsp = 0x00007fffe432d938 rip = 0x0000000012c44790
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - 47 dalvik-main space (deleted) + 0x2e288
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - rsp = 0x00007fffe432d958 rip = 0x0000000012c2e288
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - 48 dalvik-zygote space (deleted) + 0x91b88
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - rsp = 0x00007fffe432d960 rip = 0x0000000075da7b88
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - 49 dalvik-main space (deleted) + 0x2ed48
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - rsp = 0x00007fffe432d968 rip = 0x0000000012c2ed48
[task 2019-12-17T04:52:14.909Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - 50 dalvik-jit-code-cache (deleted) + 0x2d651
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - rsp = 0x00007fffe432d970 rip = 0x00007635aabd3651
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - 51 dalvik-LinearAlloc (deleted) + 0x591e0
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - rsp = 0x00007fffe432d978 rip = 0x00007635bf66c1e0
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - 52 dalvik-main space (deleted) + 0x2ed48
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - rsp = 0x00007fffe432d980 rip = 0x0000000012c2ed48
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - 53 dalvik-main space (deleted) + 0x44f20
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - rsp = 0x00007fffe432d988 rip = 0x0000000012c44f20
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - 54 boot-framework.oat + 0x1de0e8e
[task 2019-12-17T04:52:14.910Z] 04:52:14 INFO - rsp = 0x00007fffe432d990 rip = 0x0000000074d33e8e
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - 55 system@framework@boot-framework.art + 0x3e1420
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - rsp = 0x00007fffe432d998 rip = 0x0000000071681420
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - 56 dalvik-LinearAlloc (deleted) + 0x52e30
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - rsp = 0x00007fffe432d9a0 rip = 0x00007635bf665e30
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - 57 dalvik-main space (deleted) + 0xa8f790
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - rsp = 0x00007fffe432d9a8 rip = 0x000000001368f790
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - 58 dalvik-main space (deleted) + 0xa953d0
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - rsp = 0x00007fffe432d9b0 rip = 0x00000000136953d0
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - 59 dalvik-main space (deleted) + 0x44c00
[task 2019-12-17T04:52:14.911Z] 04:52:14 INFO - rsp = 0x00007fffe432d9b8 rip = 0x0000000012c44c00
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - 60 dalvik-main space (deleted) + 0x44f20
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - rsp = 0x00007fffe432d9c8 rip = 0x0000000012c44f20
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - 61 dalvik-jit-code-cache (deleted) + 0x2e205
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - rsp = 0x00007fffe432d9d0 rip = 0x00007635aabd4205
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - 62 dalvik-LinearAlloc (deleted) + 0x52e30
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - rsp = 0x00007fffe432d9d8 rip = 0x00007635bf665e30
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - 63 dalvik-main space (deleted) + 0x1c2fe0
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - rsp = 0x00007fffe432d9f8 rip = 0x0000000012dc2fe0
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - 64 libart.so + 0x138c83
[task 2019-12-17T04:52:14.912Z] 04:52:14 INFO - rsp = 0x00007fffe432da00 rip = 0x00007635c0a44c83
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - 65 system@framework@boot.art + 0x50f2c8
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - rsp = 0x00007fffe432da08 rip = 0x0000000070fbe2c8
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - 66 dalvik-main space (deleted) + 0xa8f790
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - rsp = 0x00007fffe432da30 rip = 0x000000001368f790
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - 67 dalvik-main space (deleted) + 0x1c24a0
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - rsp = 0x00007fffe432da38 rip = 0x0000000012dc24a0
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - 68 dalvik-LinearAlloc (deleted) + 0x52ed8
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - rsp = 0x00007fffe432da40 rip = 0x00007635bf665ed8
[task 2019-12-17T04:52:14.913Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - 69 dalvik-main space (deleted) + 0x8e3ca0
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - rsp = 0x00007fffe432da48 rip = 0x00000000134e3ca0
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - 70 dalvik-main space (deleted) + 0x660400
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - rsp = 0x00007fffe432da50 rip = 0x0000000013260400
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - 71 dalvik-main space (deleted) + 0x1c2fe0
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - rsp = 0x00007fffe432da58 rip = 0x0000000012dc2fe0
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - 72 dalvik-jit-code-cache (deleted) + 0x4b90
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - rsp = 0x00007fffe432da60 rip = 0x00007635aabaab90
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - 73 dalvik-LinearAlloc (deleted) + 0x52ed8
[task 2019-12-17T04:52:14.914Z] 04:52:14 INFO - rsp = 0x00007fffe432da68 rip = 0x00007635bf665ed8
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - 74 dalvik-main space (deleted) + 0x22200
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - rsp = 0x00007fffe432daa0 rip = 0x0000000012c22200
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - 75 dalvik-main space (deleted) + 0x74f820
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - rsp = 0x00007fffe432dab0 rip = 0x000000001334f820
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - 76 dalvik-main space (deleted) + 0xaa33c0
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - rsp = 0x00007fffe432dab8 rip = 0x00000000136a33c0
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - 77 dalvik-LinearAlloc (deleted) + 0x3fa10
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - rsp = 0x00007fffe432dac0 rip = 0x00007635bf652a10
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.915Z] 04:52:14 INFO - 78 dalvik-main space (deleted) + 0xaa33c0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - rsp = 0x00007fffe432dac8 rip = 0x00000000136a33c0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - 79 dalvik-main space (deleted) + 0x282c0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - rsp = 0x00007fffe432dad0 rip = 0x0000000012c282c0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - 80 dalvik-main space (deleted) + 0x19b0a0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - rsp = 0x00007fffe432dad8 rip = 0x0000000012d9b0a0
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - 81 boot-framework.oat + 0x20b38e3
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - rsp = 0x00007fffe432dae0 rip = 0x00000000750068e3
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.916Z] 04:52:14 INFO - 82 system@framework@boot-framework.art + 0x3ca618
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - rsp = 0x00007fffe432dae8 rip = 0x000000007166a618
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - 83 dalvik-main space (deleted) + 0x86fdd8
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - rsp = 0x00007fffe432daf0 rip = 0x000000001346fdd8
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - 84 dalvik-main space (deleted) + 0x5ec000
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - rsp = 0x00007fffe432daf8 rip = 0x00000000131ec000
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - 85 dalvik-LinearAlloc (deleted) + 0x3fa10
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - rsp = 0x00007fffe432db00 rip = 0x00007635bf652a10
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - 86 dalvik-main space (deleted) + 0xa8f790
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - rsp = 0x00007fffe432db08 rip = 0x000000001368f790
[task 2019-12-17T04:52:14.917Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - 87 dalvik-main space (deleted) + 0x282c0
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - rsp = 0x00007fffe432db10 rip = 0x0000000012c282c0
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - 88 dalvik-main space (deleted) + 0x19b0a0
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - rsp = 0x00007fffe432db18 rip = 0x0000000012d9b0a0
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - 89 dalvik-jit-code-cache (deleted) + 0x240a
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - rsp = 0x00007fffe432db20 rip = 0x00007635aaba840a
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - 90 dalvik-LinearAlloc (deleted) + 0x3fa10
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - rsp = 0x00007fffe432db28 rip = 0x00007635bf652a10
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.918Z] 04:52:14 INFO - 91 libart.so + 0x609a14
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - rsp = 0x00007fffe432db70 rip = 0x00007635c0f15a14
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - 92 libart.so + 0x13fc2a
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - rsp = 0x00007fffe432db80 rip = 0x00007635c0a4bc2a
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - 93 system@framework@boot.art + 0x50f2c8
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - rsp = 0x00007fffe432db88 rip = 0x0000000070fbe2c8
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - 94 dalvik-main space (deleted) + 0x86fdd8
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - rsp = 0x00007fffe432dbb0 rip = 0x000000001346fdd8
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - 95 dalvik-main space (deleted) + 0x19b320
[task 2019-12-17T04:52:14.919Z] 04:52:14 INFO - rsp = 0x00007fffe432dbb8 rip = 0x0000000012d9b320
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - 96 dalvik-main space (deleted) + 0x68bf90
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - rsp = 0x00007fffe432dbc0 rip = 0x000000001328bf90
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - 97 dalvik-main space (deleted) + 0x40e440
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - rsp = 0x00007fffe432dbc8 rip = 0x000000001300e440
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - 98 dalvik-main space (deleted) + 0x19b0a0
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - rsp = 0x00007fffe432dbd0 rip = 0x0000000012d9b0a0
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.920Z] 04:52:14 INFO - 99 dalvik-main space (deleted) + 0x68bfa0
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - rsp = 0x00007fffe432dbd8 rip = 0x000000001328bfa0
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - 100 dalvik-jit-code-cache (deleted) + 0x51509
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - rsp = 0x00007fffe432dbe0 rip = 0x00007635aabf7509
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - 101 dalvik-LinearAlloc (deleted) + 0x538
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - rsp = 0x00007fffe432dbe8 rip = 0x00007635bf613538
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - 102 dalvik-main space (deleted) + 0x19b320
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - rsp = 0x00007fffe432dc10 rip = 0x0000000012d9b320
[task 2019-12-17T04:52:14.921Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - 103 dalvik-main space (deleted) + 0x68bc70
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - rsp = 0x00007fffe432dc18 rip = 0x000000001328bc70
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - 104 dalvik-main space (deleted) + 0x57c180
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - rsp = 0x00007fffe432dc20 rip = 0x000000001317c180
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - 105 system@framework@boot.art + 0x3412c8
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - rsp = 0x00007fffe432dc28 rip = 0x0000000070df02c8
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - 106 system@framework@boot.art + 0x1fd518
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - rsp = 0x00007fffe432dc30 rip = 0x0000000070cac518
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.922Z] 04:52:14 INFO - 107 system@framework@boot.art + 0x206668
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - rsp = 0x00007fffe432dc38 rip = 0x0000000070cb5668
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - 108 dalvik-jit-code-cache (deleted) + 0x221b0
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - rsp = 0x00007fffe432dc40 rip = 0x00007635aabc81b0
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - 109 dalvik-LinearAlloc (deleted) + 0xd2c8
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - rsp = 0x00007fffe432dc48 rip = 0x00007635c55042c8
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - 110 dalvik-main space (deleted) + 0x68bfc0
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - rsp = 0x00007fffe432dc50 rip = 0x000000001328bfc0
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.923Z] 04:52:14 INFO - 111 dalvik-main space (deleted) + 0x57c280
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - rsp = 0x00007fffe432dc58 rip = 0x000000001317c280
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - 112 dalvik-main space (deleted) + 0x68bc70
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - rsp = 0x00007fffe432dc60 rip = 0x000000001328bc70
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - 113 dalvik-main space (deleted) + 0x57c180
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - rsp = 0x00007fffe432dc68 rip = 0x000000001317c180
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - 114 boot-framework.oat + 0x1657220
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - rsp = 0x00007fffe432dc70 rip = 0x00000000745aa220
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.924Z] 04:52:14 INFO - 115 system@framework@boot-framework.art + 0x256538
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - rsp = 0x00007fffe432dc78 rip = 0x00000000714f6538
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - 116 dalvik-main space (deleted) + 0x68bc70
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - rsp = 0x00007fffe432dc90 rip = 0x000000001328bc70
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - 117 dalvik-main space (deleted) + 0x35100
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - rsp = 0x00007fffe432dc98 rip = 0x0000000012c35100
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - 118 boot-framework.oat + 0x1657296
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - rsp = 0x00007fffe432dca0 rip = 0x00000000745aa296
[task 2019-12-17T04:52:14.925Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - 119 system@framework@boot-framework.art + 0x256570
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - rsp = 0x00007fffe432dca8 rip = 0x00000000714f6570
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - 120 dalvik-main space (deleted) + 0x95d8
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - rsp = 0x00007fffe432dcb8 rip = 0x0000000012c095d8
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - 121 dalvik-main space (deleted) + 0x350c0
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - rsp = 0x00007fffe432dcc0 rip = 0x0000000012c350c0
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - 122 dalvik-main space (deleted) + 0x57c180
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - rsp = 0x00007fffe432dcd0 rip = 0x000000001317c180
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - 123 dalvik-main space (deleted) + 0x350c0
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - rsp = 0x00007fffe432dcd8 rip = 0x0000000012c350c0
[task 2019-12-17T04:52:14.928Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 124 boot-framework.oat + 0x1e12c44
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dce0 rip = 0x0000000074d65c44
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 125 system@framework@boot-framework.art + 0x32c2f0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dce8 rip = 0x00000000715cc2f0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 126 dalvik-main space (deleted) + 0x95d8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dcf0 rip = 0x0000000012c095d8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 127 system@framework@boot.art + 0x3405c8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd18 rip = 0x0000000070def5c8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 128 dalvik-main space (deleted) + 0x35100
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd30 rip = 0x0000000012c35100
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 129 dalvik-main space (deleted) + 0x57c180
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd40 rip = 0x000000001317c180
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 130 system@framework@boot-framework.art + 0x419fd8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd60 rip = 0x00000000716b9fd8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 131 boot-framework.oat + 0x165ee53
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd70 rip = 0x00000000745b1e53
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 132 system@framework@boot-framework.art + 0x419fd8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd78 rip = 0x00000000716b9fd8
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 133 dalvik-main space (deleted) + 0x22e40
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd80 rip = 0x0000000012c22e40
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 134 dalvik-main space (deleted) + 0x133d0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd88 rip = 0x0000000012c133d0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 135 dalvik-main space (deleted) + 0x22e10
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd90 rip = 0x0000000012c22e10
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 136 system@framework@boot-framework.art + 0x41c0b0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dd98 rip = 0x00000000716bc0b0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 137 boot-framework.oat + 0x165e9dc
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dda0 rip = 0x00000000745b19dc
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 138 system@framework@boot-framework.art + 0x419fa0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432dda8 rip = 0x00000000716b9fa0
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - 139 system@framework@boot-framework.art + 0x41ac50
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - rsp = 0x00007fffe432ddc0 rip = 0x00000000716bac50
[task 2019-12-17T04:52:14.929Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - 140 dalvik-main space (deleted) + 0x35100
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - rsp = 0x00007fffe432ddc8 rip = 0x0000000012c35100
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - 141 dalvik-main space (deleted) + 0x22d40
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - rsp = 0x00007fffe432ddd0 rip = 0x0000000012c22d40
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - 142 system@framework@boot-framework.art + 0x41ac50
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - rsp = 0x00007fffe432ddd8 rip = 0x00000000716bac50
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - 143 dalvik-main space (deleted) + 0x35100
[task 2019-12-17T04:52:14.930Z] 04:52:14 INFO - rsp = 0x00007fffe432dde0 rip = 0x0000000012c35100
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - 144 dalvik-main space (deleted) + 0x2b208
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - rsp = 0x00007fffe432dde8 rip = 0x0000000012c2b208
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - 145 system@framework@boot-framework.art + 0x3d388
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - rsp = 0x00007fffe432ddf0 rip = 0x00000000712dd388
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - 146 dalvik-main space (deleted) + 0x36100
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - rsp = 0x00007fffe432ddf8 rip = 0x0000000012c36100
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - 147 boot-framework.oat + 0x166da50
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - rsp = 0x00007fffe432de00 rip = 0x00000000745c0a50
[task 2019-12-17T04:52:14.931Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - 148 system@framework@boot-framework.art + 0x41ac50
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - rsp = 0x00007fffe432de08 rip = 0x00000000716bac50
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - 149 dalvik-main space (deleted) + 0x36100
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - rsp = 0x00007fffe432de10 rip = 0x0000000012c36100
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - Found by: stack scanning
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - 150 boot-framework.oat + 0x6fa69f
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - rsp = 0x00007fffe432de50 rip = 0x000000007364d69f
[task 2019-12-17T04:52:14.932Z] 04:52:14 INFO - Found by: stack scanning

Flags: needinfo?(etoop)
Whiteboard: [geckoview:m1912][retriggered][stockwell fixed:patch] → [geckoview:m1912][retriggered][stockwell needswork:owner]

This has reached the disable-recommended queue.

James can you take a look?

Flags: needinfo?(snorp)

Hi James, have you had a chance to take a look?

Flags: needinfo?(snorp)
Flags: needinfo?(snorp)
Whiteboard: [geckoview:m1912][retriggered][stockwell disable-recommended] → [geckoview:m1912][retriggered][stockwell needswork:owner]
Whiteboard: [geckoview:m1912][retriggered][stockwell disable-recommended] → [geckoview:m1912][retriggered][stockwell needswork:owner]
Flags: needinfo?(etoop)
Whiteboard: [geckoview:m1912][retriggered][stockwell needswork:owner] → [geckoview:m1912][geckoview:m74][retriggered][stockwell needswork:owner]

This bug is describing a hang in the test harness after a test has been successfully run. Further investigation into the cause of the test harness hang is needed.

Assignee: nobody → snorp
Flags: needinfo?(snorp)

The majority of these seem to be caused by this:

[task 2020-01-07T23:39:38.127Z] 23:39:38     INFO -  TEST-START | org.mozilla.geckoview.test.crash.ParentCrashTest.crashParent
[task 2020-01-07T23:39:44.842Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: numtests=621
[task 2020-01-07T23:39:44.842Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stream=.
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: test=crashParent
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: class=org.mozilla.geckoview.test.crash.ParentCrashTest
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: current=621
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  org.mozilla.geckoview.test | INSTRUMENTATION_STATUS_CODE: 0
[task 2020-01-07T23:39:44.843Z] 23:39:44     INFO -  TEST-PASS | org.mozilla.geckoview.test.crash.ParentCrashTest.crashParent | took 6718ms
[task 2020-01-08T00:13:12.950Z] 00:13:12  WARNING -  TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds
[task 2020-01-08T00:13:12.950Z] 00:13:12     INFO -  Passed: 547
[task 2020-01-08T00:13:12.950Z] 00:13:12     INFO -  Failed: 0
[task 2020-01-08T00:13:12.951Z] 00:13:12     INFO -  Todo: 74
[task 2020-01-08T00:13:12.951Z] 00:13:12     INFO -  SUITE-END | took 2401s
[task 2020-01-08T00:13:13.578Z] 00:13:13     INFO -  Stopping web server
[task 2020-01-08T00:13:13.582Z] 00:13:13     INFO -  Stopping web socket server
[task 2020-01-08T00:13:13.602Z] 00:13:13     INFO -  Stopping ssltunnel

So ParentCrashTest.crashParent() is finishing, but then the instrumentation runner gets confused about something. I don't know if this is a bug in the runner or we are leaving something in a bad state. We might try just upgrading to androidx.test, since the stuff we're using from the support library is EOL now.

It looks like this went away (or we have a more specific signature). I'll close this one.

Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Priority: P1 → P3
Assignee: snorp → nobody
Severity: normal → S3

Update:

There have been 30 failures within the last 7 days, all of them on Android 7.0 x86-64 WebRender debug/ debug-isolated-process.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=414827662&repo=autoland&lineNumber=20983

[task 2023-05-06T02:59:43.698Z] 02:59:43     INFO -  TEST-START | org.mozilla.geckoview.test.WebPushTest#subscribeNullDelegate
[task 2023-05-06T02:59:44.230Z] 02:59:44  WARNING -  TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds
[task 2023-05-06T02:59:44.230Z] 02:59:44     INFO -  Passed: 884
[task 2023-05-06T02:59:44.230Z] 02:59:44     INFO -  Failed: 0
[task 2023-05-06T02:59:44.230Z] 02:59:44     INFO -  Todo: 138
[task 2023-05-06T02:59:44.230Z] 02:59:44  WARNING -  TEST-UNEXPECTED-FAIL | runjunit.py | Some tests did not run (probably due to a crash in the harness)
[task 2023-05-06T02:59:44.230Z] 02:59:44     INFO -  SUITE-END | took 2401s

:owlish as the owner of this component, can you assign this to someone?

Flags: needinfo?(bugzeeeeee)
Whiteboard: [geckoview:m1912][geckoview:m74][retriggered][stockwell unknown] → [geckoview:m1912][geckoview:m74][retriggered][stockwell needswork:owner]

I reviewed several recent failures. None of them were hung and I saw no unusual delays. These are just tasks that are running a bit slower; increasing the max-run-time seems like the best way forward.

Flags: needinfo?(bugzeeeeee)

The task max-run-time is already 3600. Keeping this timeout less than the
task max-run-time enables cleaner failures for hangs. Hopefully 3000 s will
be enough.

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/881f3696f1ab
Increase junit max-run-time to 3000 seconds r=jmaher
Summary: Intermittent TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 2400 seconds → Intermittent TEST-UNEXPECTED-TIMEOUT | runjunit.py | Timed out after 3000 seconds
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: