Closed Bug 1474758 Opened 6 years ago Closed 5 years ago

Intermittent Android 7.0 jsreftest | application ran for longer than allowed maximum time

Categories

(Firefox for Android Graveyard :: Testing, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure, Whiteboard: [TV-bf nowfound])

Crash Data

Filed by: csabou [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=187465082&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/Km0D60HRQw2UjXEygjKxiA/runs/0/artifacts/public/logs/live_backing.log https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Km0D60HRQw2UjXEygjKxiA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 [task 2018-07-10T22:58:13.770Z] 22:58:13 INFO - REFTEST PROCESS-CRASH | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=non262/Function/regress-85880.js | application crashed [@ libc.so + 0x1c3dc] [task 2018-07-10T22:58:13.771Z] 22:58:13 INFO - Crash dump filename: /tmp/tmpa3Mj0B/0b4430cf-a86f-88c3-e69e-9a26312f055a.dmp [task 2018-07-10T22:58:13.771Z] 22:58:13 INFO - Operating system: Android [task 2018-07-10T22:58:13.773Z] 22:58:13 INFO - 0.0.0 Linux 2.6.29-gea477bb #1 Wed Sep 26 11:04:45 PDT 2012 armv7l [task 2018-07-10T22:58:13.773Z] 22:58:13 INFO - CPU: arm [task 2018-07-10T22:58:13.774Z] 22:58:13 INFO - ARMv7 ARM Cortex-A8 features: swp,half,thumb,fastmult,vfpv2,edsp,neon,vfpv3 [task 2018-07-10T22:58:13.775Z] 22:58:13 INFO - 1 CPU [task 2018-07-10T22:58:13.775Z] 22:58:13 INFO - GPU: UNKNOWN [task 2018-07-10T22:58:13.776Z] 22:58:13 INFO - Crash reason: SIGABRT [task 2018-07-10T22:58:13.776Z] 22:58:13 INFO - Crash address: 0x15b9 [task 2018-07-10T22:58:13.777Z] 22:58:13 INFO - Process uptime: not available [task 2018-07-10T22:58:13.778Z] 22:58:13 INFO - Thread 0 (crashed) [task 2018-07-10T22:58:13.779Z] 22:58:13 INFO - 0 libc.so + 0x1c3dc [task 2018-07-10T22:58:13.779Z] 22:58:13 INFO - r0 = 0xfffffffc r1 = 0xbe9fd4a8 r2 = 0x00000010 r3 = 0xffffffff [task 2018-07-10T22:58:13.780Z] 22:58:13 INFO - r4 = 0x2a03f470 r5 = 0x00000000 r6 = 0x2a03f484 r7 = 0x000000fc [task 2018-07-10T22:58:13.781Z] 22:58:13 INFO - r8 = 0x2a03f4b8 r9 = 0x00000014 r10 = 0x2a00d0a0 r12 = 0xbe9fd4a8 [task 2018-07-10T22:58:13.782Z] 22:58:13 INFO - fp = 0xbe9fd61c sp = 0xbe9fd488 lr = 0x400b1fb5 pc = 0x400433dc [task 2018-07-10T22:58:13.783Z] 22:58:13 INFO - Found by: given as instruction pointer in context
Severity: critical → normal
Crash Signature: [@ libc.so + 0x1c3dc]
Keywords: crash
Summary: Intermittent http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=non262/Function/regress-85880.js | application crashed [@ libc.so + 0x1c3dc] → Intermittent http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=non262/Function/regress-85880.js | application ran for longer than allowed maximum time
Summary: Intermittent http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=non262/Function/regress-85880.js | application ran for longer than allowed maximum time → Intermittent Android jsreftest | application ran for longer than allowed maximum time
(In reply to Joel Maher ( :jmaher ) (UTC+2) from comment #1) > test was not found: It can now be backfilled on current revisions. See bug 1483421. https://treeherder.mozilla.org/logviewer.html#?job_id=194167180&repo=try
Whiteboard: [TV-bf notfound] → [TV-bf nowfound]
Some of the recent failures are Android 7.0 and are caused by bad hardware - "machine-16" - being discussed in bug 1509433.
See Also: → 1509433

Recent failures have been mostly android-em-7-0-x86_64, and most of those are on specific hosts; trying to sort that out in bug 1540280.

See Also: → 1540280

Over the last 7 days there are 34 present on this bug. These happen on android-em-7-0-x86_64

Here is the most recent log failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=240862272&repo=autoland&lineNumber=18990

[task 2019-04-17T08:32:13.666Z] 08:32:13 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-falsey.js
[task 2019-04-17T08:32:13.667Z] 08:32:13 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-falsey.js | 2880 / 4352 (66%)
[task 2019-04-17T08:32:24.504Z] 08:32:24 INFO - REFTEST TEST-PASS | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-falsey.js | item 1
[task 2019-04-17T08:32:24.504Z] 08:32:24 INFO - REFTEST TEST-END | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-falsey.js
[task 2019-04-17T08:32:24.507Z] 08:32:24 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-negative.js
[task 2019-04-17T08:32:24.509Z] 08:32:24 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-negative.js | 2881 / 4352 (66%)
[task 2019-04-17T08:32:24.514Z] 08:32:24 INFO - REFTEST TEST-PASS | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-negative.js | item 1
[task 2019-04-17T08:32:24.514Z] 08:32:24 INFO - REFTEST TEST-END | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-negative.js
[task 2019-04-17T08:32:24.515Z] 08:32:24 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-positive.js
[task 2019-04-17T08:32:24.516Z] 08:32:24 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-positive.js | 2882 / 4352 (66%)
[task 2019-04-17T08:32:24.520Z] 08:32:24 INFO - REFTEST TEST-PASS | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-positive.js | item 1
[task 2019-04-17T08:32:24.521Z] 08:32:24 INFO - REFTEST TEST-END | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-positive.js
[task 2019-04-17T08:32:24.522Z] 08:32:24 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js
[task 2019-04-17T08:32:24.524Z] 08:32:24 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js | 2883 / 4352 (66%)
[task 2019-04-17T08:32:24.527Z] 08:32:24 INFO - REFTEST TEST-PASS | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js | item 1
[task 2019-04-17T08:32:24.528Z] 08:32:24 INFO - REFTEST TEST-END | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js
[task 2019-04-17T08:32:24.529Z] 08:32:24 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-04-17T08:32:24.837Z] 08:32:24 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-04-17T08:32:24.839Z] 08:32:24 INFO - REFTEST TEST-INFO | started process screentopng
[task 2019-04-17T08:32:26.785Z] 08:32:26 INFO - REFTEST TEST-INFO | screentopng: exit 0
[task 2019-04-17T08:32:42.116Z] 08:32:42 WARNING - TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js | application ran for longer than allowed maximum time
[task 2019-04-17T08:32:42.118Z] 08:32:42 INFO - remoteautomation.py | Application ran for: 1:50:43.988924
[task 2019-04-17T08:32:43.371Z] 08:32:43 INFO - REFTEST INFO | Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpdeylOl/23e631a3-4ce7-8bae-676f-ff6bc8678324.dmp /builds/worker/workspace/build/symbols
[task 2019-04-17T08:33:06.478Z] 08:33:06 INFO - REFTEST INFO | Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/23e631a3-4ce7-8bae-676f-ff6bc8678324.dmp
[task 2019-04-17T08:33:06.479Z] 08:33:06 INFO - REFTEST INFO | Saved app info as /builds/worker/workspace/build/blobber_upload_dir/23e631a3-4ce7-8bae-676f-ff6bc8678324.extra
[task 2019-04-17T08:33:06.508Z] 08:33:06 INFO - REFTEST PROCESS-CRASH | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/annexB/built-ins/String/prototype/substr/length-to-int-err.js | application crashed [@ libc.so + 0x8c66a]
[task 2019-04-17T08:33:06.508Z] 08:33:06 INFO - Crash dump filename: /tmp/tmpdeylOl/23e631a3-4ce7-8bae-676f-ff6bc8678324.dmp
[task 2019-04-17T08:33:06.509Z] 08:33:06 INFO - Operating system: Android
[task 2019-04-17T08:33:06.510Z] 08:33:06 INFO - 0.0.0 Linux 3.10.0+ #1 PREEMPT Thu Jan 5 00:46:30 UTC 2017 x86_64
[task 2019-04-17T08:33:06.510Z] 08:33:06 INFO - CPU: amd64
[task 2019-04-17T08:33:06.511Z] 08:33:06 INFO - family 6 model 2 stepping 3
[task 2019-04-17T08:33:06.512Z] 08:33:06 INFO - 1 CPU

Flags: needinfo?(sdetar)

Jason, could you help me triage this bug and understand if it is a priority for us to look at.

Flags: needinfo?(sdetar) → needinfo?(jorendorff)

Recent android-em-7-0-x86_64 timeouts are all on machine-7, machine-12, and machine-20. The same tasks complete in less than half that time normally.

:coop - Does that make any sense in light of your recent work on bug 1540280? Anything more you can do?

Flags: needinfo?(coop)

(In reply to Geoff Brown [:gbrown] from comment #39)

Recent android-em-7-0-x86_64 timeouts are all on machine-7, machine-12, and machine-20. The same tasks complete in less than half that time normally.

:coop - Does that make any sense in light of your recent work on bug 1540280? Anything more you can do?

All those instances have been re-created from scratch since Friday, so it is unlikely to be an issue with the image itself.

One thing I would remind people about is that we run 4 workers per instance in packet.net. If the workloads themselves have changed at all recently, e.g. by consuming more resources etc, we may be running into resource contention/starvation from shared tenancy.

Adding a NI for Wander in case he has more insight. In particular, is there a way to tell which workers were running on a single instance concurrently?

Flags: needinfo?(coop) → needinfo?(wcosta)
Depends on: 1545308

(In reply to Chris Cooper [:coop] pronoun: he from comment #40)

(In reply to Geoff Brown [:gbrown] from comment #39)

Recent android-em-7-0-x86_64 timeouts are all on machine-7, machine-12, and machine-20. The same tasks complete in less than half that time normally.

:coop - Does that make any sense in light of your recent work on bug 1540280? Anything more you can do?

All those instances have been re-created from scratch since Friday, so it is
unlikely to be an issue with the image itself.

One thing I would remind people about is that we run 4 workers per instance
in packet.net. If the workloads themselves have changed at all recently,
e.g. by consuming more resources etc, we may be running into resource
contention/starvation from shared tenancy.

Adding a NI for Wander in case he has more insight. In particular, is there
a way to tell which workers were running on a single instance concurrently?

Yes, that's right, we may be facing resource starvation in shared tasks. The solution would be to reduce worker capacity and spawn more instances.

Flags: needinfo?(wcosta)

More duplicates over in bug 1544536.

This isn't a JS engine bug. Moving to Taskcluster::General, like bug 1540280.

Component: JavaScript Engine → General
Flags: needinfo?(jorendorff)
Priority: P5 → --
Product: Core → Taskcluster

There are 67 total failures in the last 7 days on android-em-7-0-x86_64 debug and opt, on machines 7, 12, 20,23

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=241671710&repo=autoland&lineNumber=19386

[task 2019-04-21T09:08:33.971Z] 09:08:33 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/class/elements/syntax/early-errors/grammar-class-body-ctor-duplicate.js;error=SyntaxError
[task 2019-04-21T09:08:33.972Z] 09:08:33 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/class/elements/syntax/early-errors/grammar-class-body-ctor-duplicate.js;error=SyntaxError | 4814 / 10339 (46%)
[task 2019-04-21T09:08:33.973Z] 09:08:33 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-04-21T09:08:34.232Z] 09:08:34 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-04-21T09:08:34.232Z] 09:08:34 INFO - REFTEST TEST-INFO | started process screentopng
[task 2019-04-21T09:08:35.741Z] 09:08:35 INFO - REFTEST TEST-INFO | screentopng: exit 0
[task 2019-04-21T09:08:50.750Z] 09:08:50 WARNING - TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/class/elements/syntax/early-errors/grammar-class-body-ctor-duplicate.js;error=SyntaxError | application ran for longer than allowed maximum time
[task 2019-04-21T09:08:50.751Z] 09:08:50 INFO - remoteautomation.py | Application ran for: 1:50:41.418349
[task 2019-04-21T09:08:51.779Z] 09:08:51 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/DAY31kIETi-lomPvQ-ui1Q/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-04-21T09:08:59.721Z] 09:08:59 INFO - REFTEST INFO | Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpTkvHYV/51c03b74-d894-6b0d-56e5-181d0f59f260.dmp /tmp/tmp3CafBI
[task 2019-04-21T09:09:12.312Z] 09:09:12 INFO - REFTEST INFO | Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/51c03b74-d894-6b0d-56e5-181d0f59f260.dmp
[task 2019-04-21T09:09:12.313Z] 09:09:12 INFO - REFTEST INFO | Saved app info as /builds/worker/workspace/build/blobber_upload_dir/51c03b74-d894-6b0d-56e5-181d0f59f260.extra
[task 2019-04-21T09:09:12.328Z] 09:09:12 INFO - REFTEST PROCESS-CRASH | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/class/elements/syntax/early-errors/grammar-class-body-ctor-duplicate.js;error=SyntaxError | application crashed [@ libc.so + 0x8c66a]
[task 2019-04-21T09:09:12.329Z] 09:09:12 INFO - Crash dump filename: /tmp/tmpTkvHYV/51c03b74-d894-6b0d-56e5-181d0f59f260.dmp
[task 2019-04-21T09:09:12.330Z] 09:09:12 INFO - Operating system: Android
[task 2019-04-21T09:09:12.331Z] 09:09:12 INFO - 0.0.0 Linux 3.10.0+ #1 PREEMPT Thu Jan 5 00:46:30 UTC 2017 x86_64
[task 2019-04-21T09:09:12.331Z] 09:09:12 INFO - CPU: amd64
[task 2019-04-21T09:09:12.331Z] 09:09:12 INFO - family 6 model 2 stepping 3
[task 2019-04-21T09:09:12.331Z] 09:09:12 INFO - 1 CPU
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - GPU: UNKNOWN
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - Crash reason: SIGABRT
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - Crash address: 0x110a
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - Process uptime: not available
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - Thread 0 (crashed)
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - 0 libc.so + 0x8c66a
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-04-21T09:09:12.333Z] 09:09:12 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000ffffffff
[task 2019-04-21T09:09:12.334Z] 09:09:12 INFO - rsi = 0x00007fff42759200 rdi = 0x000000000000001f
[task 2019-04-21T09:09:12.334Z] 09:09:12 INFO - rbp = 0x00000000ffffffff rsp = 0x00007fff42759188
[task 2019-04-21T09:09:12.334Z] 09:09:12 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-04-21T09:09:12.335Z] 09:09:12 INFO - r10 = 0x00000000ffffffff r11 = 0x0000000000000246
[task 2019-04-21T09:09:12.335Z] 09:09:12 INFO - r12 = 0x00000000ffffffff r13 = 0x0000759801929f40
[task 2019-04-21T09:09:12.336Z] 09:09:12 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-04-21T09:09:12.336Z] 09:09:12 INFO - rip = 0x00007598042bd66a
[task 2019-04-21T09:09:12.336Z] 09:09:12 INFO - Found by: given as instruction pointer in context
[task 2019-04-21T09:09:12.337Z] 09:09:12 INFO - 1 libc.so + 0x2a0e6
[task 2019-04-21T09:09:12.337Z] 09:09:12 INFO - rsp = 0x00007fff42759190 rip = 0x000075980425b0e6
[task 2019-04-21T09:09:12.337Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.338Z] 09:09:12 INFO - 2 libutils.so + 0x184e7
[task 2019-04-21T09:09:12.338Z] 09:09:12 INFO - rsp = 0x00007fff427591b0 rip = 0x0000759805a134e7
[task 2019-04-21T09:09:12.338Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.339Z] 09:09:12 INFO - 3 system@framework@boot-framework.art + 0x3d5350
[task 2019-04-21T09:09:12.339Z] 09:09:12 INFO - rsp = 0x00007fff427591d8 rip = 0x0000000071675350
[task 2019-04-21T09:09:12.339Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.340Z] 09:09:12 INFO - 4 boot-framework.oat + 0x1000
[task 2019-04-21T09:09:12.340Z] 09:09:12 INFO - rsp = 0x00007fff427591e8 rip = 0x0000000072f54000
[task 2019-04-21T09:09:12.340Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.341Z] 09:09:12 INFO - 5 dalvik-main space (deleted) + 0x17f20
[task 2019-04-21T09:09:12.341Z] 09:09:12 INFO - rsp = 0x00007fff42759200 rip = 0x0000000012c17f20
[task 2019-04-21T09:09:12.341Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.342Z] 09:09:12 INFO - 6 libart.so + 0x2d39f9
[task 2019-04-21T09:09:12.342Z] 09:09:12 INFO - rsp = 0x00007fff42759220 rip = 0x00007598013df9f9
[task 2019-04-21T09:09:12.342Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.343Z] 09:09:12 INFO - 7 boot-framework.oat + 0x15bc3f0
[task 2019-04-21T09:09:12.343Z] 09:09:12 INFO - rsp = 0x00007fff42759228 rip = 0x000000007450f3f0
[task 2019-04-21T09:09:12.343Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.344Z] 09:09:12 INFO - 8 libart.so + 0x4f1be9
[task 2019-04-21T09:09:12.344Z] 09:09:12 INFO - rsp = 0x00007fff42759230 rip = 0x00007598015fdbe9
[task 2019-04-21T09:09:12.345Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.345Z] 09:09:12 INFO - 9 system@framework@boot-framework.art + 0x2a9f48
[task 2019-04-21T09:09:12.346Z] 09:09:12 INFO - rsp = 0x00007fff42759248 rip = 0x0000000071549f48
[task 2019-04-21T09:09:12.346Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.346Z] 09:09:12 INFO - 10 system@framework@boot-framework.art + 0x3d5350
[task 2019-04-21T09:09:12.347Z] 09:09:12 INFO - rsp = 0x00007fff42759260 rip = 0x0000000071675350
[task 2019-04-21T09:09:12.347Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.347Z] 09:09:12 INFO - 11 libart.so + 0x39c37f
[task 2019-04-21T09:09:12.347Z] 09:09:12 INFO - rsp = 0x00007fff427592a0 rip = 0x00007598014a837f
[task 2019-04-21T09:09:12.348Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.348Z] 09:09:12 INFO - 12 libart.so + 0x6e3c40
[task 2019-04-21T09:09:12.348Z] 09:09:12 INFO - rsp = 0x00007fff427592a8 rip = 0x00007598017efc40
[task 2019-04-21T09:09:12.349Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.349Z] 09:09:12 INFO - 13 dalvik-allocspace main rosalloc space 1 mark-bitmap 2 (deleted) + 0x3f4000
[task 2019-04-21T09:09:12.350Z] 09:09:12 INFO - rsp = 0x00007fff427592b8 rip = 0x0000759800000000
[task 2019-04-21T09:09:12.350Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.350Z] 09:09:12 INFO - 14 libutils.so + 0x1837b
[task 2019-04-21T09:09:12.351Z] 09:09:12 INFO - rsp = 0x00007fff42759300 rip = 0x0000759805a1337b
[task 2019-04-21T09:09:12.351Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.351Z] 09:09:12 INFO - 15 system@framework@boot-framework.art + 0x3f5b10
[task 2019-04-21T09:09:12.351Z] 09:09:12 INFO - rsp = 0x00007fff42759328 rip = 0x0000000071695b10
[task 2019-04-21T09:09:12.351Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.352Z] 09:09:12 INFO - 16 dalvik-main space (deleted) + 0x95d8
[task 2019-04-21T09:09:12.352Z] 09:09:12 INFO - rsp = 0x00007fff42759330 rip = 0x0000000012c095d8
[task 2019-04-21T09:09:12.352Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.352Z] 09:09:12 INFO - 17 libandroid_runtime.so + 0xff856
[task 2019-04-21T09:09:12.353Z] 09:09:12 INFO - rsp = 0x00007fff42759340 rip = 0x0000759804f9d856
[task 2019-04-21T09:09:12.353Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.354Z] 09:09:12 INFO - 18 system@framework@boot-framework.art + 0x3f5b10
[task 2019-04-21T09:09:12.354Z] 09:09:12 INFO - rsp = 0x00007fff42759348 rip = 0x0000000071695b10
[task 2019-04-21T09:09:12.355Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.355Z] 09:09:12 INFO - 19 boot-framework.oat + 0x1e17220
[task 2019-04-21T09:09:12.356Z] 09:09:12 INFO - rsp = 0x00007fff42759360 rip = 0x0000000074d6a220
[task 2019-04-21T09:09:12.356Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.357Z] 09:09:12 INFO - 20 system@framework@boot-framework.art + 0x3f5b10
[task 2019-04-21T09:09:12.357Z] 09:09:12 INFO - rsp = 0x00007fff42759378 rip = 0x0000000071695b10
[task 2019-04-21T09:09:12.358Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.359Z] 09:09:12 INFO - 21 dalvik-main space (deleted) + 0x95d8
[task 2019-04-21T09:09:12.359Z] 09:09:12 INFO - rsp = 0x00007fff42759380 rip = 0x0000000012c095d8
[task 2019-04-21T09:09:12.360Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.361Z] 09:09:12 INFO - 22 system@framework@boot-framework.art + 0x3f5790
[task 2019-04-21T09:09:12.361Z] 09:09:12 INFO - rsp = 0x00007fff42759388 rip = 0x0000000071695790
[task 2019-04-21T09:09:12.361Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.361Z] 09:09:12 INFO - 23 dalvik-allocspace main rosalloc space 1 mark-bitmap 2 (deleted) + 0x3f4007
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - rsp = 0x00007fff427593a0 rip = 0x0000759800000007
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - 24 system@framework@boot.art + 0x50f2c8
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - rsp = 0x00007fff427593a8 rip = 0x0000000070fbe2c8
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - 25 system@framework@boot-framework.art + 0x3f5b10
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - rsp = 0x00007fff427593f0 rip = 0x0000000071695b10
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - 26 dalvik-main space (deleted) + 0x95d8
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - rsp = 0x00007fff427593f8 rip = 0x0000000012c095d8
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.362Z] 09:09:12 INFO - 27 boot-framework.oat + 0x1e19034
[task 2019-04-21T09:09:12.363Z] 09:09:12 INFO - rsp = 0x00007fff42759400 rip = 0x0000000074d6c034
[task 2019-04-21T09:09:12.363Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.363Z] 09:09:12 INFO - 28 system@framework@boot-framework.art + 0x3f5b10
[task 2019-04-21T09:09:12.364Z] 09:09:12 INFO - rsp = 0x00007fff42759408 rip = 0x0000000071695b10
[task 2019-04-21T09:09:12.364Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.364Z] 09:09:12 INFO - 29 system@framework@boot.art + 0x342bf0
[task 2019-04-21T09:09:12.365Z] 09:09:12 INFO - rsp = 0x00007fff42759428 rip = 0x0000000070df1bf0
[task 2019-04-21T09:09:12.365Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.365Z] 09:09:12 INFO - 30 system@framework@boot.art + 0x1fd518
[task 2019-04-21T09:09:12.366Z] 09:09:12 INFO - rsp = 0x00007fff42759450 rip = 0x0000000070cac518
[task 2019-04-21T09:09:12.366Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.366Z] 09:09:12 INFO - 31 dalvik-main space (deleted) + 0x116340
[task 2019-04-21T09:09:12.367Z] 09:09:12 INFO - rsp = 0x00007fff42759470 rip = 0x0000000012d16340
[task 2019-04-21T09:09:12.367Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.367Z] 09:09:12 INFO - 32 dalvik-main space (deleted) + 0x1410c0
[task 2019-04-21T09:09:12.368Z] 09:09:12 INFO - rsp = 0x00007fff42759478 rip = 0x0000000012d410c0
[task 2019-04-21T09:09:12.368Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.369Z] 09:09:12 INFO - 33 dalvik-main space (deleted) + 0x1410c0
[task 2019-04-21T09:09:12.369Z] 09:09:12 INFO - rsp = 0x00007fff42759480 rip = 0x0000000012d410c0
[task 2019-04-21T09:09:12.370Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.370Z] 09:09:12 INFO - 34 system@framework@boot-framework.art + 0x3ad778
[task 2019-04-21T09:09:12.370Z] 09:09:12 INFO - rsp = 0x00007fff42759488 rip = 0x000000007164d778
[task 2019-04-21T09:09:12.370Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.371Z] 09:09:12 INFO - 35 boot-framework.oat + 0x1e160c4
[task 2019-04-21T09:09:12.371Z] 09:09:12 INFO - rsp = 0x00007fff427594c0 rip = 0x0000000074d690c4
[task 2019-04-21T09:09:12.371Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.372Z] 09:09:12 INFO - 36 system@framework@boot-framework.art + 0x39bd40
[task 2019-04-21T09:09:12.372Z] 09:09:12 INFO - rsp = 0x00007fff427594c8 rip = 0x000000007163bd40
[task 2019-04-21T09:09:12.372Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.372Z] 09:09:12 INFO - 37 dalvik-main space (deleted) + 0x95d8
[task 2019-04-21T09:09:12.373Z] 09:09:12 INFO - rsp = 0x00007fff427594d8 rip = 0x0000000012c095d8
[task 2019-04-21T09:09:12.373Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.374Z] 09:09:12 INFO - 38 dalvik-main space (deleted) + 0x17f20
[task 2019-04-21T09:09:12.374Z] 09:09:12 INFO - rsp = 0x00007fff427594e0 rip = 0x0000000012c17f20
[task 2019-04-21T09:09:12.374Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.374Z] 09:09:12 INFO - 39 system@framework@boot.art + 0x3412c8
[task 2019-04-21T09:09:12.374Z] 09:09:12 INFO - rsp = 0x00007fff427594e8 rip = 0x0000000070df02c8
[task 2019-04-21T09:09:12.375Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.375Z] 09:09:12 INFO - 40 system@framework@boot.art + 0x1fd518
[task 2019-04-21T09:09:12.375Z] 09:09:12 INFO - rsp = 0x00007fff427594f0 rip = 0x0000000070cac518
[task 2019-04-21T09:09:12.376Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.376Z] 09:09:12 INFO - 41 system@framework@boot.art + 0x206668
[task 2019-04-21T09:09:12.376Z] 09:09:12 INFO - rsp = 0x00007fff427594f8 rip = 0x0000000070cb5668
[task 2019-04-21T09:09:12.377Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.377Z] 09:09:12 INFO - 42 boot-framework.oat + 0x1e12ab0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759500 rip = 0x0000000074d65ab0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - 43 system@framework@boot-framework.art + 0x32c2f0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759508 rip = 0x00000000715cc2f0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - 44 dalvik-main space (deleted) + 0x95d8
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759510 rip = 0x0000000012c095d8
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - 45 system@framework@boot.art + 0x3405c8
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759538 rip = 0x0000000070def5c8
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - 46 dalvik-main space (deleted) + 0x10b6a0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759550 rip = 0x0000000012d0b6a0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - 47 dalvik-main space (deleted) + 0x1410c0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - rsp = 0x00007fff42759560 rip = 0x0000000012d410c0
[task 2019-04-21T09:09:12.381Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.382Z] 09:09:12 INFO - 48 system@framework@boot-framework.art + 0x419fd8
[task 2019-04-21T09:09:12.382Z] 09:09:12 INFO - rsp = 0x00007fff42759580 rip = 0x00000000716b9fd8
[task 2019-04-21T09:09:12.382Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.382Z] 09:09:12 INFO - 49 boot-framework.oat + 0x165ee53
[task 2019-04-21T09:09:12.383Z] 09:09:12 INFO - rsp = 0x00007fff42759590 rip = 0x00000000745b1e53
[task 2019-04-21T09:09:12.383Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.383Z] 09:09:12 INFO - 50 system@framework@boot-framework.art + 0x419fd8
[task 2019-04-21T09:09:12.384Z] 09:09:12 INFO - rsp = 0x00007fff42759598 rip = 0x00000000716b9fd8
[task 2019-04-21T09:09:12.384Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.385Z] 09:09:12 INFO - 51 dalvik-main space (deleted) + 0x29e40
[task 2019-04-21T09:09:12.385Z] 09:09:12 INFO - rsp = 0x00007fff427595a0 rip = 0x0000000012c29e40
[task 2019-04-21T09:09:12.385Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.386Z] 09:09:12 INFO - 52 dalvik-main space (deleted) + 0x133d0
[task 2019-04-21T09:09:12.386Z] 09:09:12 INFO - rsp = 0x00007fff427595a8 rip = 0x0000000012c133d0
[task 2019-04-21T09:09:12.386Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.386Z] 09:09:12 INFO - 53 dalvik-main space (deleted) + 0x29e10
[task 2019-04-21T09:09:12.387Z] 09:09:12 INFO - rsp = 0x00007fff427595b0 rip = 0x0000000012c29e10
[task 2019-04-21T09:09:12.387Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.387Z] 09:09:12 INFO - 54 system@framework@boot-framework.art + 0x41c0b0
[task 2019-04-21T09:09:12.388Z] 09:09:12 INFO - rsp = 0x00007fff427595b8 rip = 0x00000000716bc0b0
[task 2019-04-21T09:09:12.388Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.388Z] 09:09:12 INFO - 55 boot-framework.oat + 0x165e9dc
[task 2019-04-21T09:09:12.389Z] 09:09:12 INFO - rsp = 0x00007fff427595c0 rip = 0x00000000745b19dc
[task 2019-04-21T09:09:12.389Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.389Z] 09:09:12 INFO - 56 system@framework@boot-framework.art + 0x419fa0
[task 2019-04-21T09:09:12.390Z] 09:09:12 INFO - rsp = 0x00007fff427595c8 rip = 0x00000000716b9fa0
[task 2019-04-21T09:09:12.390Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.390Z] 09:09:12 INFO - 57 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.390Z] 09:09:12 INFO - rsp = 0x00007fff427595e0 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.391Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.391Z] 09:09:12 INFO - 58 dalvik-main space (deleted) + 0x17f60
[task 2019-04-21T09:09:12.392Z] 09:09:12 INFO - rsp = 0x00007fff427595e8 rip = 0x0000000012c17f60
[task 2019-04-21T09:09:12.392Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.392Z] 09:09:12 INFO - 59 dalvik-main space (deleted) + 0x29d40
[task 2019-04-21T09:09:12.392Z] 09:09:12 INFO - rsp = 0x00007fff427595f0 rip = 0x0000000012c29d40
[task 2019-04-21T09:09:12.393Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.393Z] 09:09:12 INFO - 60 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.394Z] 09:09:12 INFO - rsp = 0x00007fff427595f8 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.394Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.394Z] 09:09:12 INFO - 61 dalvik-main space (deleted) + 0x17f60
[task 2019-04-21T09:09:12.394Z] 09:09:12 INFO - rsp = 0x00007fff42759600 rip = 0x0000000012c17f60
[task 2019-04-21T09:09:12.395Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.395Z] 09:09:12 INFO - 62 dalvik-main space (deleted) + 0x31598
[task 2019-04-21T09:09:12.395Z] 09:09:12 INFO - rsp = 0x00007fff42759608 rip = 0x0000000012c31598
[task 2019-04-21T09:09:12.396Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.396Z] 09:09:12 INFO - 63 system@framework@boot-framework.art + 0x3d388
[task 2019-04-21T09:09:12.396Z] 09:09:12 INFO - rsp = 0x00007fff42759610 rip = 0x00000000712dd388
[task 2019-04-21T09:09:12.397Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.397Z] 09:09:12 INFO - 64 dalvik-main space (deleted) + 0x3b100
[task 2019-04-21T09:09:12.397Z] 09:09:12 INFO - rsp = 0x00007fff42759618 rip = 0x0000000012c3b100
[task 2019-04-21T09:09:12.398Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.398Z] 09:09:12 INFO - 65 boot-framework.oat + 0x166da50
[task 2019-04-21T09:09:12.398Z] 09:09:12 INFO - rsp = 0x00007fff42759620 rip = 0x00000000745c0a50
[task 2019-04-21T09:09:12.398Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.399Z] 09:09:12 INFO - 66 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.399Z] 09:09:12 INFO - rsp = 0x00007fff42759628 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.399Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.399Z] 09:09:12 INFO - 67 dalvik-main space (deleted) + 0x3b100
[task 2019-04-21T09:09:12.400Z] 09:09:12 INFO - rsp = 0x00007fff42759630 rip = 0x0000000012c3b100
[task 2019-04-21T09:09:12.400Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.400Z] 09:09:12 INFO - 68 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.401Z] 09:09:12 INFO - rsp = 0x00007fff42759670 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.401Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.402Z] 09:09:12 INFO - 69 libart.so + 0x137597
[task 2019-04-21T09:09:12.402Z] 09:09:12 INFO - rsp = 0x00007fff42759680 rip = 0x0000759801243597
[task 2019-04-21T09:09:12.402Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.403Z] 09:09:12 INFO - 70 dalvik-main space (deleted) + 0x29d40
[task 2019-04-21T09:09:12.403Z] 09:09:12 INFO - rsp = 0x00007fff42759690 rip = 0x0000000012c29d40
[task 2019-04-21T09:09:12.403Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.404Z] 09:09:12 INFO - 71 boot-framework.oat + 0x5c44c8
[task 2019-04-21T09:09:12.404Z] 09:09:12 INFO - rsp = 0x00007fff42759698 rip = 0x00000000735174c8
[task 2019-04-21T09:09:12.404Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.404Z] 09:09:12 INFO - 72 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.405Z] 09:09:12 INFO - rsp = 0x00007fff427596a8 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.405Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.405Z] 09:09:12 INFO - 73 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.406Z] 09:09:12 INFO - rsp = 0x00007fff427596c8 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.406Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.407Z] 09:09:12 INFO - 74 libart.so + 0x14546d
[task 2019-04-21T09:09:12.407Z] 09:09:12 INFO - rsp = 0x00007fff427596e0 rip = 0x000075980125146d
[task 2019-04-21T09:09:12.408Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.408Z] 09:09:12 INFO - 75 dalvik-main space (deleted) + 0x9560
[task 2019-04-21T09:09:12.408Z] 09:09:12 INFO - rsp = 0x00007fff42759708 rip = 0x0000000012c09560
[task 2019-04-21T09:09:12.409Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.409Z] 09:09:12 INFO - 76 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.410Z] 09:09:12 INFO - rsp = 0x00007fff42759740 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.410Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.411Z] 09:09:12 INFO - 77 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.411Z] 09:09:12 INFO - rsp = 0x00007fff42759758 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.411Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.411Z] 09:09:12 INFO - 78 libart.so + 0x4d60e8
[task 2019-04-21T09:09:12.411Z] 09:09:12 INFO - rsp = 0x00007fff42759770 rip = 0x00007598015e20e8
[task 2019-04-21T09:09:12.412Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.412Z] 09:09:12 INFO - 79 dalvik-main space (deleted) + 0x9560
[task 2019-04-21T09:09:12.412Z] 09:09:12 INFO - rsp = 0x00007fff42759778 rip = 0x0000000012c09560
[task 2019-04-21T09:09:12.412Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.413Z] 09:09:12 INFO - 80 system@framework@boot.art + 0xe2dd8
[task 2019-04-21T09:09:12.413Z] 09:09:12 INFO - rsp = 0x00007fff42759788 rip = 0x0000000070b91dd8
[task 2019-04-21T09:09:12.413Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.413Z] 09:09:12 INFO - 81 dalvik-main space (deleted) + 0x29d50
[task 2019-04-21T09:09:12.414Z] 09:09:12 INFO - rsp = 0x00007fff42759790 rip = 0x0000000012c29d50
[task 2019-04-21T09:09:12.414Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.414Z] 09:09:12 INFO - 82 system@framework@boot.art + 0x4e1000
[task 2019-04-21T09:09:12.414Z] 09:09:12 INFO - rsp = 0x00007fff427597a8 rip = 0x0000000070f90000
[task 2019-04-21T09:09:12.415Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.415Z] 09:09:12 INFO - 83 boot-framework.oat + 0x24c7309
[task 2019-04-21T09:09:12.415Z] 09:09:12 INFO - rsp = 0x00007fff427597b8 rip = 0x000000007541a309
[task 2019-04-21T09:09:12.416Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.416Z] 09:09:12 INFO - 84 boot-framework.oat + 0x24c6a80
[task 2019-04-21T09:09:12.417Z] 09:09:12 INFO - rsp = 0x00007fff427597c0 rip = 0x0000000075419a80
[task 2019-04-21T09:09:12.418Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.418Z] 09:09:12 INFO - 85 libart.so + 0x621dd3
[task 2019-04-21T09:09:12.418Z] 09:09:12 INFO - rsp = 0x00007fff427597d0 rip = 0x000075980172ddd3
[task 2019-04-21T09:09:12.418Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.419Z] 09:09:12 INFO - 86 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.419Z] 09:09:12 INFO - rsp = 0x00007fff42759818 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.420Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.420Z] 09:09:12 INFO - 87 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.420Z] 09:09:12 INFO - rsp = 0x00007fff42759820 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.421Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.421Z] 09:09:12 INFO - 88 boot-framework.oat + 0x5c44c8
[task 2019-04-21T09:09:12.421Z] 09:09:12 INFO - rsp = 0x00007fff42759840 rip = 0x00000000735174c8
[task 2019-04-21T09:09:12.422Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.422Z] 09:09:12 INFO - 89 dalvik-zygote space (deleted) + 0x5c8
[task 2019-04-21T09:09:12.422Z] 09:09:12 INFO - rsp = 0x00007fff427598a8 rip = 0x0000000075d165c8
[task 2019-04-21T09:09:12.423Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.423Z] 09:09:12 INFO - 90 system@framework@boot.art + 0x340848
[task 2019-04-21T09:09:12.423Z] 09:09:12 INFO - rsp = 0x00007fff427598b8 rip = 0x0000000070def848
[task 2019-04-21T09:09:12.423Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.424Z] 09:09:12 INFO - 91 libart.so + 0x4d0f5a
[task 2019-04-21T09:09:12.424Z] 09:09:12 INFO - rsp = 0x00007fff427598e0 rip = 0x00007598015dcf5a
[task 2019-04-21T09:09:12.424Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.424Z] 09:09:12 INFO - 92 dalvik-main space (deleted) + 0x9560
[task 2019-04-21T09:09:12.425Z] 09:09:12 INFO - rsp = 0x00007fff42759938 rip = 0x0000000012c09560
[task 2019-04-21T09:09:12.425Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.425Z] 09:09:12 INFO - 93 libart.so + 0x50742d
[task 2019-04-21T09:09:12.426Z] 09:09:12 INFO - rsp = 0x00007fff42759950 rip = 0x000075980161342d
[task 2019-04-21T09:09:12.426Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.426Z] 09:09:12 INFO - 94 boot-framework.oat + 0x6fa69f
[task 2019-04-21T09:09:12.426Z] 09:09:12 INFO - rsp = 0x00007fff42759970 rip = 0x000000007364d69f
[task 2019-04-21T09:09:12.427Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.427Z] 09:09:12 INFO - 95 dalvik-main space (deleted) + 0x29d60
[task 2019-04-21T09:09:12.427Z] 09:09:12 INFO - rsp = 0x00007fff42759978 rip = 0x0000000012c29d60
[task 2019-04-21T09:09:12.428Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.428Z] 09:09:12 INFO - 96 system@framework@boot-framework.art + 0x41ac50
[task 2019-04-21T09:09:12.428Z] 09:09:12 INFO - rsp = 0x00007fff42759980 rip = 0x00000000716bac50
[task 2019-04-21T09:09:12.428Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.429Z] 09:09:12 INFO - 97 system@framework@boot-framework.art + 0x3d388
[task 2019-04-21T09:09:12.429Z] 09:09:12 INFO - rsp = 0x00007fff42759990 rip = 0x00000000712dd388
[task 2019-04-21T09:09:12.429Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.429Z] 09:09:12 INFO - 98 libart.so + 0x4d81d5
[task 2019-04-21T09:09:12.430Z] 09:09:12 INFO - rsp = 0x00007fff427599a0 rip = 0x00007598015e41d5
[task 2019-04-21T09:09:12.430Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.430Z] 09:09:12 INFO - 99 boot-framework.oat + 0x24c872d
[task 2019-04-21T09:09:12.430Z] 09:09:12 INFO - rsp = 0x00007fff427599b8 rip = 0x000000007541b72d
[task 2019-04-21T09:09:12.431Z] 09:09:12 INFO - Found by: stack scanning
[task 2019-04-21T09:09:12.431Z] 09:09:12 INFO - 100 system@framework@boot-framework.art + 0x322f00
[task 2019-04-21T09:09:12.432Z] 09:09:12 INFO - rsp = 0x00007fff427599d0 rip = 0x00000000715c2f00
[task 2019-04-21T09:09:12.432Z] 09:09:12 INFO - Found by: stack scanning

Whiteboard: [TV-bf nowfound] → [TV-bf nowfound][stockwell needswork]
Summary: Intermittent Android jsreftest | application ran for longer than allowed maximum time → Intermittent Android 7.0 jsreftest | application ran for longer than allowed maximum time
Whiteboard: [TV-bf nowfound][stockwell disable-recommended] → [TV-bf nowfound][stockwell needswork]

There are 76 failures for this bug in the past 7 weeks happening on android-em-7-0-x86_64.

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=243217990&repo=mozilla-inbound&lineNumber=27638

Log snippet:

[task 2019-04-29T00:41:24.659Z] 00:41:24 INFO - REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/for-await-of/async-gen-dstr-let-ary-ptrn-elem-obj-prop-id.js
[task 2019-04-29T00:41:24.660Z] 00:41:24 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/for-await-of/async-gen-dstr-let-ary-ptrn-elem-obj-prop-id.js | 7099 / 10339 (68%)
[task 2019-04-29T00:41:24.660Z] 00:41:24 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-04-29T00:41:25.068Z] 00:41:25 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-04-29T00:41:25.070Z] 00:41:25 INFO - REFTEST TEST-INFO | started process screentopng
[task 2019-04-29T00:41:26.605Z] 00:41:26 INFO - REFTEST TEST-INFO | screentopng: exit 0
[task 2019-04-29T00:41:41.833Z] 00:41:41 WARNING - TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/for-await-of/async-gen-dstr-let-ary-ptrn-elem-obj-prop-id.js | application ran for longer than allowed maximum time
[task 2019-04-29T00:41:41.834Z] 00:41:41 INFO - remoteautomation.py | Application ran for: 1:50:37.879342
[task 2019-04-29T00:41:42.888Z] 00:41:42 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/G7VYjfqgRCO45dlBP78zVg/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-04-29T00:41:51.291Z] 00:41:51 INFO - REFTEST INFO | Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpN8VW8B/622b12d6-20eb-1ce1-a497-2c6f23d0f707.dmp /tmp/tmp3q3gYX
[task 2019-04-29T00:42:05.112Z] 00:42:05 INFO - REFTEST INFO | Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/622b12d6-20eb-1ce1-a497-2c6f23d0f707.dmp
[task 2019-04-29T00:42:05.113Z] 00:42:05 INFO - REFTEST INFO | Saved app info as /builds/worker/workspace/build/blobber_upload_dir/622b12d6-20eb-1ce1-a497-2c6f23d0f707.extra
[task 2019-04-29T00:42:05.124Z] 00:42:05 INFO - REFTEST PROCESS-CRASH | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=test262/language/statements/for-await-of/async-gen-dstr-let-ary-ptrn-elem-obj-prop-id.js | application crashed [@ libc.so + 0x8c66a]
[task 2019-04-29T00:42:05.125Z] 00:42:05 INFO - Crash dump filename: /tmp/tmpN8VW8B/622b12d6-20eb-1ce1-a497-2c6f23d0f707.dmp
[task 2019-04-29T00:42:05.125Z] 00:42:05 INFO - Operating system: Android
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - 0.0.0 Linux 3.10.0+ #1 PREEMPT Thu Jan 5 00:46:30 UTC 2017 x86_64
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - CPU: amd64
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - family 6 model 2 stepping 3
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - 1 CPU
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - GPU: UNKNOWN
[task 2019-04-29T00:42:05.126Z] 00:42:05 INFO - Crash reason: SIGABRT
[task 2019-04-29T00:42:05.127Z] 00:42:05 INFO - Crash address: 0x111a
[task 2019-04-29T00:42:05.127Z] 00:42:05 INFO - Process uptime: not available
[task 2019-04-29T00:42:05.127Z] 00:42:05 INFO - Thread 0 (crashed)
[task 2019-04-29T00:42:05.127Z] 00:42:05 INFO - 0 libc.so + 0x8c66a
[task 2019-04-29T00:42:05.128Z] 00:42:05 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-04-29T00:42:05.128Z] 00:42:05 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000ffffffff
[task 2019-04-29T00:42:05.128Z] 00:42:05 INFO - rsi = 0x00007fff6790efe0 rdi = 0x000000000000001f
[task 2019-04-29T00:42:05.129Z] 00:42:05 INFO - rbp = 0x00000000ffffffff rsp = 0x00007fff6790ef68
[task 2019-04-29T00:42:05.129Z] 00:42:05 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-04-29T00:42:05.129Z] 00:42:05 INFO - r10 = 0x00000000ffffffff r11 = 0x0000000000000246
[task 2019-04-29T00:42:05.130Z] 00:42:05 INFO - r12 = 0x00000000ffffffff r13 = 0x00007a656d129f40
[task 2019-04-29T00:42:05.130Z] 00:42:05 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-04-29T00:42:05.131Z] 00:42:05 INFO - rip = 0x00007a656ddb466a
[task 2019-04-29T00:42:05.131Z] 00:42:05 INFO - Found by: given as instruction pointer in context
[task 2019-04-29T00:42:05.131Z] 00:42:05 INFO - 1 libc.so + 0x2a0e6
[task 2019-04-29T00:42:05.131Z] 00:42:05 INFO - rsp = 0x00007fff6790ef70 rip = 0x00007a656dd520e6
[task 2019-04-29T00:42:05.132Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.132Z] 00:42:05 INFO - 2 libutils.so + 0x184e7
[task 2019-04-29T00:42:05.132Z] 00:42:05 INFO - rsp = 0x00007fff6790ef90 rip = 0x00007a656fcf84e7
[task 2019-04-29T00:42:05.133Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.133Z] 00:42:05 INFO - 3 boot-framework.oat + 0x1000
[task 2019-04-29T00:42:05.133Z] 00:42:05 INFO - rsp = 0x00007fff6790efc8 rip = 0x0000000072f54000
[task 2019-04-29T00:42:05.134Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.134Z] 00:42:05 INFO - 4 dalvik-main space (deleted) + 0x2f0c0
[task 2019-04-29T00:42:05.134Z] 00:42:05 INFO - rsp = 0x00007fff6790efe0 rip = 0x0000000012c2f0c0
[task 2019-04-29T00:42:05.135Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.135Z] 00:42:05 INFO - 5 libart.so + 0x2d39f9
[task 2019-04-29T00:42:05.135Z] 00:42:05 INFO - rsp = 0x00007fff6790f000 rip = 0x00007a656cbdf9f9
[task 2019-04-29T00:42:05.135Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.136Z] 00:42:05 INFO - 6 boot-framework.oat + 0x15bc3f0
[task 2019-04-29T00:42:05.136Z] 00:42:05 INFO - rsp = 0x00007fff6790f008 rip = 0x000000007450f3f0
[task 2019-04-29T00:42:05.136Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.137Z] 00:42:05 INFO - 7 libart.so + 0x4f1be9
[task 2019-04-29T00:42:05.137Z] 00:42:05 INFO - rsp = 0x00007fff6790f010 rip = 0x00007a656cdfdbe9
[task 2019-04-29T00:42:05.138Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.138Z] 00:42:05 INFO - 8 libart.so + 0x39c37f
[task 2019-04-29T00:42:05.138Z] 00:42:05 INFO - rsp = 0x00007fff6790f030 rip = 0x00007a656cca837f
[task 2019-04-29T00:42:05.138Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.139Z] 00:42:05 INFO - 9 libart.so + 0x6e3c40
[task 2019-04-29T00:42:05.139Z] 00:42:05 INFO - rsp = 0x00007fff6790f038 rip = 0x00007a656cfefc40
[task 2019-04-29T00:42:05.139Z] 00:42:05 INFO - Found by: stack scanning
[task 2019-04-29T00:42:05.139Z] 00:42:05 INFO - 10 libart.so + 0x39c37f
[task 2019-04-29T00:42:05.140Z] 00:42:05 INFO - rsp = 0x00007fff6790f080 rip = 0x00007a656cca837f
[task 2019-04-29T00:42:05.140Z] 00:42:05 INFO - Found by: stack scanning

This isn't a Taskcluster issue -- judging by the error message the test has some timeout in it that is being exceeded. Jason says it's not a JS bug, so off to Firefox :: General for triage.

Product: Taskcluster → Firefox
Whiteboard: [TV-bf nowfound][stockwell disable-recommended] → [TV-bf nowfound][stockwell needswork:owner]

Given that it's Android, it's most certainly not Firefox General.

If it's "not a Taskcluster issue" that on certain machines a test chunk which normally runs in 30 minutes sometimes takes more than two hours, with individual tests which ordinarily take less than a second to complete taking more than ten seconds to complete, then I guess it's up to Fennec Testing to write a test to run as a preflight that will determine whether or not it is running in an emulator that is capable of doing work, and to set RETRY on the job and abort it if it is not.

Component: General → Testing
Product: Firefox → Firefox for Android

..or find the race condition or missed signal that causes this, or fix the bug in the emulator that causes this :)

Virtually all of the recent failures here show the conditions of bug 1545308: /proc/cpuinfo of the host reports drastically reduced MHz, even before the emulator is started.

Summary: Intermittent Android 7.0 jsreftest | application ran for longer than allowed maximum time → Intermittent Android 7.0 (or Android 4.3) jsreftest | application ran for longer than allowed maximum time

I don't see any Android 4.3 failures reported in this bug for the last 7 days; any such failures definitely have a different cause than the current Android 7.0 ones, which depend on bug 1545308. Also, there's not usually any connection between exceeding the maximum application run time and crashes or no-output timeouts.

See Also: 1509433

Changing the summary back; maybe a different problem is causing these intermittent Android 4.3 timeouts. The CPU appears to be 2500MHz on those machines.

See Also: 1476746
Summary: Intermittent Android 7.0 (or Android 4.3) jsreftest | application ran for longer than allowed maximum time → Intermittent Android 7.0 jsreftest | application ran for longer than allowed maximum time
Crash Signature: [@ libc.so + 0x8c66a]
Whiteboard: [TV-bf nowfound][stockwell disable-recommended] → [TV-bf nowfound]
Crash Signature: [@ libc.so + 0x8c66a] → [@ libc.so + 0x8c66a] [@ libc.so + 0x1cc48]
Crash Signature: [@ libc.so + 0x8c66a] [@ libc.so + 0x1cc48] → [@ libc.so + 0x8c66a] [@ libc.so + 0x1cc48]
Whiteboard: [TV-bf nowfound][stockwell disable-recommended] → [TV-bf nowfound]

Efforts to resolve this continue in bug 1545308.

The priority flag is not set for this bug.
:gbrown, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gbrown)
Flags: needinfo?(gbrown)
Priority: -- → P3
Depends on: 1552334
Whiteboard: [TV-bf nowfound][stockwell disable-recommended] → [TV-bf nowfound]

Failures have been practically eliminated by the workaround landed in bug 1552334.

Recent failures reported here are misclassifications.

Assignee: nobody → gbrown
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.