1537745 - Perma Android 8.0 TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: Audio/Video, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=234921156&repo=autoland

https://queue.taskcluster.net/v1/task/GvQL_ExbRWq6EYzFhUvgPg/runs/0/artifacts/public/logs/live_backing.log

03:49:02 INFO - 352 INFO TEST-START | dom/media/webaudio/test/test_waveShaperPassThrough.html
03:49:02 INFO - 353 INFO TEST-OK | dom/media/webaudio/test/test_waveShaperPassThrough.html | took 381ms
03:49:02 INFO - 354 INFO TEST-START | Shutdown
03:49:02 INFO - 355 INFO Passed: 4285
03:49:02 INFO - 356 INFO Failed: 0
03:49:02 INFO - 357 INFO Todo: 48
03:49:02 INFO - 358 INFO Mode: non-e10s
03:49:02 INFO - 359 INFO Slowest: 8230ms - /tests/dom/media/webaudio/test/test_bug1027864.html
03:49:02 INFO - 360 INFO SimpleTest FINISHED
03:49:05 INFO - wait for org.mozilla.fennec_aurora complete; top activity=com.bitbar.testdroid.monitor
03:49:05 INFO - remoteautomation.py | Application ran for: 0:02:21.085878
03:49:06 INFO - adb Ignoring attempt to chmod external storage
03:49:06 INFO - runtests.py | Running with scheme: https
03:49:06 INFO - runtests.py | Running with e10s: False
03:49:06 INFO - runtests.py | Running with serviceworker_e10s: False
03:49:06 INFO - runtests.py | Running with socketprocess_e10s: False
03:49:06 INFO - runtests.py | Running tests: start.
03:49:07 INFO - remoteautomation.py | runApp deleted /sdcard/tests/logs/mochitest.log
03:49:08 INFO - adb Granting important runtime permissions to org.mozilla.fennec_aurora
03:49:10 INFO - adb launch_application: am start -W -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es env9 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es env8 R_LOG_DESTINATION=stderr --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 DISABLE_UNSAFE_CPOW_WARNINGS=1 --es env2 R_LOG_VERBOSE=1 --es env1 XPCOM_DEBUG_BREAK=stack --es env0 MOZ_CRASHREPORTER=1 --es env7 MOZ_LOG_FILE=/sdcard/tests/mozlog/moz.log --es env6 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 MOZ_IN_AUTOMATION=1 --es env4 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env12 MOZ_HIDE_RESULTS_TABLE=1 --es env11 R_LOG_LEVEL=6 --es env10 MOZ_CRASHREPORTER_NO_REPORT=1 -d "https://example.com:443/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
03:49:11 INFO - remoteautomation.py | Application pid: 7541
03:55:41 INFO - wait for org.mozilla.fennec_aurora complete; top activity=org.mozilla.fennec_aurora
03:55:41 INFO - org.mozilla.fennec_aurora unexpectedly found running. Killing...
03:55:41 INFO - TEST-INFO | started process screencap
03:55:42 INFO - TEST-INFO | screencap: exit 0
03:55:53 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Jan-Ivar, this started to permafail for mda runs on Android 8.0 when bug 1335740 landed. Please investigate.

Comment 2

[Intermittent Failures Robot]

315 failures in 475 pushes (0.663 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-inbound: 62
• mozilla-central: 24
• autoland: 228
• try: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 124
• android-hw-p2-8-0-arm7-api-16-pgo: 188
• linux64: 1
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-21&endday=2019-03-21&tree=all

Comment 3

[Andreea Pavel [:apavel]]

There are 351 total failures in the last 2 days on android-hw-p2-8-0-arm7-api-16 debug

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235439088&repo=mozilla-central&lineNumber=2832

10:54:26 INFO - 38 INFO TEST-START | dom/media/tests/mochitest/test_selftest.html
10:54:26 INFO - 39 INFO TEST-OK | dom/media/tests/mochitest/test_selftest.html | took 1321ms
10:54:26 INFO - 40 INFO TEST-START | Shutdown
10:54:26 INFO - 41 INFO Passed: 1842
10:54:26 INFO - 42 INFO Failed: 0
10:54:26 INFO - 43 INFO Todo: 72
10:54:26 INFO - 44 INFO Mode: non-e10s
10:54:26 INFO - 45 INFO Slowest: 3976ms - /tests/dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html
10:54:26 INFO - 46 INFO SimpleTest FINISHED
10:54:28 INFO - wait for org.mozilla.fennec_aurora complete; top activity=com.bitbar.testdroid.monitor
10:54:28 INFO - remoteautomation.py | Application ran for: 0:00:57.313129
10:54:29 INFO - adb Ignoring attempt to chmod external storage
10:54:29 INFO - runtests.py | Running with scheme: https
10:54:29 INFO - runtests.py | Running with e10s: False
10:54:29 INFO - runtests.py | Running with serviceworker_e10s: False
10:54:29 INFO - runtests.py | Running with socketprocess_e10s: False
10:54:29 INFO - runtests.py | Running tests: start.
10:54:30 INFO - remoteautomation.py | runApp deleted /sdcard/tests/logs/mochitest.log
10:54:31 INFO - adb Granting important runtime permissions to org.mozilla.fennec_aurora
10:54:33 INFO - adb launch_application: am start -W -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es env9 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es env8 R_LOG_DESTINATION=stderr --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 DISABLE_UNSAFE_CPOW_WARNINGS=1 --es env2 R_LOG_VERBOSE=1 --es env1 XPCOM_DEBUG_BREAK=stack --es env0 MOZ_CRASHREPORTER=1 --es env7 MOZ_LOG_FILE=/sdcard/tests/mozlog/moz.log --es env6 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 MOZ_IN_AUTOMATION=1 --es env4 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env12 MOZ_HIDE_RESULTS_TABLE=1 --es env11 R_LOG_LEVEL=6 --es env10 MOZ_CRASHREPORTER_NO_REPORT=1 -d "https://example.com:443/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
10:54:34 INFO - remoteautomation.py | Application pid: 6290
11:01:04 INFO - wait for org.mozilla.fennec_aurora complete; top activity=org.mozilla.fennec_aurora
11:01:04 INFO - org.mozilla.fennec_aurora unexpectedly found running. Killing...
11:01:04 INFO - TEST-INFO | started process screencap
11:01:05 INFO - TEST-INFO | screencap: exit 0
11:01:16 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output
11:01:16 INFO - remoteautomation.py | Application ran for: 0:06:45.850372
11:01:17 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpLyVrp_/4a8e3dbf-a5b2-b4c3-02ab-935d5142686e.dmp /builds/worker/workspace/build/symbols
11:01:25 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/4a8e3dbf-a5b2-b4c3-02ab-935d5142686e.dmp
11:01:25 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/4a8e3dbf-a5b2-b4c3-02ab-935d5142686e.extra
11:01:25 WARNING - PROCESS-CRASH | remoteautomation.py | application crashed [@ libc.so + 0x48eb8]
11:01:25 INFO - Crash dump filename: /tmp/tmpLyVrp_/4a8e3dbf-a5b2-b4c3-02ab-935d5142686e.dmp
11:01:25 INFO - Operating system: Android
11:01:25 INFO - 0.0.0 Linux 4.4.56-g594d847d09a1 #1 SMP PREEMPT Thu Oct 26 22:34:08 UTC 2017 armv8l
11:01:25 INFO - CPU: arm
11:01:25 INFO - ARMv1 Qualcomm part(0x51008010) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
11:01:25 INFO - 8 CPUs
11:01:25 INFO - GPU: UNKNOWN
11:01:25 INFO - Crash reason: SIGABRT

Comment 4

[Cosmin Sabou [:CosminS]]

Jan, the timeouts in this bug are causing longer runtimes for the these kind of jobs http://tinyurl.com/y2srb59v and that leads to a large backlog on Android hw jobs. Some are with "Duration: Not started (queued for 813 minute(s))"
Could please take a look into this asap? The failure rate for this one is really high. Thank you.

Comment 5

[Geoff Brown [:gbrown]]

By the way, these hangs occur consistently in android-hw mda tasks whenever the https scheme is used:

10:54:29 INFO - runtests.py | Running with scheme: https
...
11:01:16 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output

I looked at several android-hw mda tasks (mda1/2/3 presumably running different tests): all of them hung on the first browser test run with https scheme.

https://bugzilla.mozilla.org/show_bug.cgi?id=1500977#c43 might be helpful: screenshots show Fennec displaying "The proxy server is refusing connections" with "https://example.com/tests?..." in the url bar, and logcats may show LOAD_ERROR + some media activity.

Comment 6

[Intermittent Failures Robot]

123 failures in 537 pushes (0.229 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 26
• autoland: 36
• mozilla-inbound: 60
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 69
• android-hw-p2-8-0-arm7-api-16-pgo: 52
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-22&endday=2019-03-22&tree=all

Comment 7

[Intermittent Failures Robot]

156 failures in 132 pushes (1.182 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 18
• autoland: 104
• mozilla-central: 21
• try: 12
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 70
• android-hw-p2-8-0-arm7-api-16-pgo: 84
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-23&endday=2019-03-23&tree=all

Comment 8

[Andreea Pavel [:apavel]]

There are 827 total failures in the last 4 days, on android-hw-p2-8-0-arm7-api-16-pgo opt, android-hw-p2-8-0-arm7-api-16
debug and android-em-7-0-x86_64 opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235740412&repo=autoland&lineNumber=2818

18:31:23 INFO - wait for org.mozilla.fennec_aurora complete; top activity=org.mozilla.fennec_aurora
18:31:23 INFO - org.mozilla.fennec_aurora unexpectedly found running. Killing...
18:31:23 INFO - TEST-INFO | started process screencap
18:31:24 INFO - TEST-INFO | screencap: exit 0
18:31:36 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output
18:31:36 INFO - remoteautomation.py | Application ran for: 0:06:45.732943
18:31:36 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/JGWSmip5TF2nZv1ZB3HtUw/artifacts/public/build/target.crashreporter-symbols.zip
18:31:40 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmp7imVC7/674dd9dc-2c0b-dea8-972c-b4663e630c42.dmp /tmp/tmpPXjvaQ
18:31:45 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/674dd9dc-2c0b-dea8-972c-b4663e630c42.dmp
18:31:45 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/674dd9dc-2c0b-dea8-972c-b4663e630c42.extra
18:31:45 WARNING - PROCESS-CRASH | remoteautomation.py | application crashed [@ libc.so + 0x48eb8]
18:31:45 INFO - Crash dump filename: /tmp/tmp7imVC7/674dd9dc-2c0b-dea8-972c-b4663e630c42.dmp
18:31:45 INFO - Operating system: Android
18:31:45 INFO - 0.0.0 Linux 4.4.56-g594d847d09a1 #1 SMP PREEMPT Thu Oct 26 22:34:08 UTC 2017 armv8l
18:31:45 INFO - CPU: arm
18:31:45 INFO - ARMv1 Qualcomm part(0x51008010) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
18:31:45 INFO - 8 CPUs
18:31:45 INFO - GPU: UNKNOWN
18:31:45 INFO - Crash reason: SIGABRT
18:31:45 INFO - Crash address: 0x1aef
18:31:45 INFO - Process uptime: not available
18:31:45 INFO - Thread 0 (crashed)
18:31:45 INFO - 0 libc.so + 0x48eb8
18:31:45 INFO - r0 = 0xfffffffc r1 = 0xffec1558 r2 = 0x00000010 r3 = 0xffffffff
18:31:45 INFO - r4 = 0x00000000 r5 = 0x00000008 r6 = 0x00000000 r7 = 0x0000015a
18:31:45 INFO - r8 = 0xffec1558 r9 = 0xf35b23fc r10 = 0x00000000 r12 = 0xffec1518
18:31:45 INFO - fp = 0xf35b23b0 sp = 0xffec1508 lr = 0xf4c2f79d pc = 0xf4c5ceb8
18:31:45 INFO - Found by: given as instruction pointer in context
18:31:45 INFO - 1 libc.so + 0x1b7cb
18:31:45 INFO - sp = 0xffec1538 pc = 0xf4c2f7cd
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 2 libutils.so + 0x1012f
18:31:45 INFO - sp = 0xffec1548 pc = 0xf6095131
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 3 binder + 0x56
18:31:45 INFO - sp = 0xffec154c pc = 0xdb3d6058
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 4 binder + 0x5e
18:31:45 INFO - sp = 0xffec1554 pc = 0xdb3d6060
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 5 libbinder.so + 0x44caf
18:31:45 INFO - sp = 0xffec1588 pc = 0xf5297cb1
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 6 dalvik-main space (region space) (deleted) + 0x3c8646
18:31:45 INFO - sp = 0xffec158c pc = 0x12fc8648
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 7 dalvik-main space (region space) (deleted) + 0x3c8646
18:31:45 INFO - sp = 0xffec159c pc = 0x12fc8648
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 8 dalvik-main space (region space) (deleted) + 0x3c8646
18:31:45 INFO - sp = 0xffec15ac pc = 0x12fc8648
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 9 dalvik-main space (region space) (deleted) + 0x3c8646
18:31:45 INFO - sp = 0xffec15b0 pc = 0x12fc8648
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 10 dalvik-main space (region space) (deleted) + 0x3c8646
18:31:45 INFO - sp = 0xffec15c8 pc = 0x12fc8648
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 11 libart.so + 0xb2d03
18:31:45 INFO - sp = 0xffec15d0 pc = 0xf3131d05
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 12 dalvik-main space (region space) (deleted) + 0x3c866e
18:31:45 INFO - sp = 0xffec15d8 pc = 0x12fc8670
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 13 libart.so + 0x4062cf
18:31:45 INFO - sp = 0xffec15e8 pc = 0xf34852d1
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 14 system@[e-mail] + 0xb7946
18:31:45 INFO - sp = 0xffec1600 pc = 0x6fdcf948
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 15 libart.so + 0xb2d03
18:31:45 INFO - sp = 0xffec1608 pc = 0xf3131d05
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 16 libart.so + 0x4062cf
18:31:45 INFO - sp = 0xffec1620 pc = 0xf34852d1
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 17 libart.so + 0x43c4ae
18:31:45 INFO - sp = 0xffec1630 pc = 0xf34bb4b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 18 libart.so + 0x3bb542
18:31:45 INFO - sp = 0xffec1634 pc = 0xf343a544
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 19 libart.so + 0xb3de1
18:31:45 INFO - sp = 0xffec1638 pc = 0xf3132de3
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 20 libart.so + 0x3bb576
18:31:45 INFO - sp = 0xffec1654 pc = 0xf343a578
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 21 libart.so + 0x3bb57e
18:31:45 INFO - sp = 0xffec1658 pc = 0xf343a580
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 22 libutils.so + 0x1001f
18:31:45 INFO - sp = 0xffec1680 pc = 0xf6095021
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 23 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec1698 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 24 libandroid_runtime.so + 0xb46a3
18:31:45 INFO - sp = 0xffec16a8 pc = 0xf3eed6a5
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 25 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec16b0 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 26 system@[e-mail] + 0x6b656
18:31:45 INFO - sp = 0xffec16b4 pc = 0x6fa18658
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 27 boot-framework.oat + 0x73a485
18:31:45 INFO - sp = 0xffec16c0 pc = 0x715f3487
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 28 boot-framework.oat + 0x5b34d9
18:31:45 INFO - sp = 0xffec16d0 pc = 0x7146c4db
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 29 system@[e-mail] + 0x388fde
18:31:45 INFO - sp = 0xffec16d4 pc = 0x700a0fe0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 30 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec16e0 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 31 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec1734 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 32 boot-framework.oat + 0xd66719
18:31:45 INFO - sp = 0xffec1740 pc = 0x71c1f71b
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 33 system@[e-mail] + 0x3891fe
18:31:45 INFO - sp = 0xffec1744 pc = 0x700a1200
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 34 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec1748 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 35 system@[e-mail] + 0x123e26
18:31:45 INFO - sp = 0xffec1758 pc = 0x6fe3be28
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 36 system@[e-mail] + 0x11f666
18:31:45 INFO - sp = 0xffec175c pc = 0x6fe37668
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 37 system@[e-mail] + 0x6b656
18:31:45 INFO - sp = 0xffec1760 pc = 0x6fa18658
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 38 dalvik-main space (region space) (deleted) + 0x925636
18:31:45 INFO - sp = 0xffec176c pc = 0x13525638
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 39 system@[e-mail] + 0x355e1a
18:31:45 INFO - sp = 0xffec17a4 pc = 0x7006de1c
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 40 system@[e-mail] + 0x1f16
18:31:45 INFO - sp = 0xffec17a8 pc = 0x6fd19f18
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 41 dalvik-main space (region space) (deleted) + 0x901bb6
18:31:45 INFO - sp = 0xffec17ac pc = 0x13501bb8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 42 system@[e-mail] + 0x6505e
18:31:45 INFO - sp = 0xffec17b0 pc = 0x6fa12060
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 43 system@[e-mail] + 0x10a28e
18:31:45 INFO - sp = 0xffec17b4 pc = 0x6fe22290
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 44 boot-framework.oat + 0xd61725
18:31:45 INFO - sp = 0xffec17c0 pc = 0x71c1a727
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 45 system@[e-mail] + 0x30f9d2
18:31:45 INFO - sp = 0xffec17c4 pc = 0x700279d4
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 46 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec17c8 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 47 dalvik-main space (region space) (deleted) + 0x901b6e
18:31:45 INFO - sp = 0xffec17d4 pc = 0x13501b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 48 system@[e-mail] + 0x16963e
18:31:45 INFO - sp = 0xffec17d8 pc = 0x6fe81640
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 49 system@[e-mail] + 0x102c0e
18:31:45 INFO - sp = 0xffec17dc pc = 0x6fe1ac10
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 50 system@[e-mail] + 0xf9a8e
18:31:45 INFO - sp = 0xffec17e0 pc = 0x6fe11a90
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 51 system@[e-mail] + 0x25b6e
18:31:45 INFO - sp = 0xffec17e4 pc = 0x6fbb7b70
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 52 system@[e-mail] + 0x109636
18:31:45 INFO - sp = 0xffec17e8 pc = 0x6fe21638
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 53 system@[e-mail] + 0x133056
18:31:45 INFO - sp = 0xffec17ec pc = 0x6fe4b058
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 54 system@[e-mail] + 0xfee3e
18:31:45 INFO - sp = 0xffec17f0 pc = 0x6fe16e40
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 55 system@[e-mail] + 0xffd9e
18:31:45 INFO - sp = 0xffec17f4 pc = 0x6fe17da0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 56 system@[e-mail] + 0x122e86
18:31:45 INFO - sp = 0xffec17f8 pc = 0x6fe3ae88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 57 system@[e-mail] + 0x116b66
18:31:45 INFO - sp = 0xffec17fc pc = 0x6fe2eb68
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 58 system@[e-mail] + 0x8b2be
18:31:45 INFO - sp = 0xffec1800 pc = 0x6fa382c0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 59 system@[e-mail] + 0x9055e
18:31:45 INFO - sp = 0xffec1804 pc = 0x6fa3d560
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 60 system@[e-mail] + 0xffc1e
18:31:45 INFO - sp = 0xffec1808 pc = 0x6fe17c20
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 61 system@[e-mail] + 0x6505e
18:31:45 INFO - sp = 0xffec1810 pc = 0x6fa12060
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 62 system@[e-mail] + 0x10a28e
18:31:45 INFO - sp = 0xffec1814 pc = 0x6fe22290
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 63 dalvik-main space (region space) (deleted) + 0x901936
18:31:45 INFO - sp = 0xffec181c pc = 0x13501938
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 64 dalvik-main space (region space) (deleted) + 0x901bb6
18:31:45 INFO - sp = 0xffec1820 pc = 0x13501bb8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 65 dalvik-main space (region space) (deleted) + 0x7896
18:31:45 INFO - sp = 0xffec1828 pc = 0x12c07898
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 66 system@[e-mail] + 0x1f16
18:31:45 INFO - sp = 0xffec182c pc = 0x6fd19f18
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 67 dalvik-main space (region space) (deleted) + 0x919a2e
18:31:45 INFO - sp = 0xffec1830 pc = 0x13519a30
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 68 dalvik-main space (region space) (deleted) + 0x7f9e
18:31:45 INFO - sp = 0xffec1854 pc = 0x12c07fa0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 69 dalvik-main space (region space) (deleted) + 0x7ff6
18:31:45 INFO - sp = 0xffec1858 pc = 0x12c07ff8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 70 system@[e-mail] + 0x6456e
18:31:45 INFO - sp = 0xffec1860 pc = 0x6fd7c570
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 71 system@[e-mail] + 0x109e7e
18:31:45 INFO - sp = 0xffec1868 pc = 0x6fe21e80
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 72 system@[e-mail] + 0x1f16
18:31:45 INFO - sp = 0xffec186c pc = 0x6fd19f18
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 73 system@[e-mail] + 0x5d4f6
18:31:45 INFO - sp = 0xffec1870 pc = 0x6fd754f8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 74 dalvik-main space (region space) (deleted) + 0x901c2e
18:31:45 INFO - sp = 0xffec1874 pc = 0x13501c30
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 75 dalvik-main space (region space) (deleted) + 0x9015d6
18:31:45 INFO - sp = 0xffec187c pc = 0x135015d8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 76 boot-framework.oat + 0x653d0b
18:31:45 INFO - sp = 0xffec1880 pc = 0x7150cd0d
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 77 system@[e-mail] + 0x378686
18:31:45 INFO - sp = 0xffec1884 pc = 0x70090688
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 78 dalvik-main space (region space) (deleted) + 0x7896
18:31:45 INFO - sp = 0xffec1888 pc = 0x12c07898
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 79 system@[e-mail] + 0xbb1ae
18:31:45 INFO - sp = 0xffec1894 pc = 0x6fa681b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 80 dalvik-main space (region space) (deleted) + 0x901bf6
18:31:45 INFO - sp = 0xffec1898 pc = 0x13501bf8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 81 dalvik-main space (region space) (deleted) + 0x73ee
18:31:45 INFO - sp = 0xffec189c pc = 0x12c073f0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 82 system@[e-mail] + 0x168ff6
18:31:45 INFO - sp = 0xffec18a0 pc = 0x6fe80ff8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 83 libart.so + 0x3dbd71
18:31:45 INFO - sp = 0xffec18d0 pc = 0xf345ad73
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 84 dalvik-main space (region space) (deleted) + 0x5bce
18:31:45 INFO - sp = 0xffec18d8 pc = 0x12c05bd0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 85 boot-framework.oat + 0x543db7
18:31:45 INFO - sp = 0xffec18dc pc = 0x713fcdb9
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 86 libart.so + 0x3e07ed
18:31:45 INFO - sp = 0xffec1900 pc = 0xf345f7ef
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 87 dalvik-main space (region space) (deleted) + 0x5bce
18:31:45 INFO - sp = 0xffec1958 pc = 0x12c05bd0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 88 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec1968 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 89 system@[e-mail] + 0x378686
18:31:45 INFO - sp = 0xffec196c pc = 0x70090688
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 90 libart.so + 0xac881
18:31:45 INFO - sp = 0xffec1988 pc = 0xf312b883
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 91 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec1990 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 92 dalvik-main space (region space) (deleted) + 0x694e
18:31:45 INFO - sp = 0xffec1994 pc = 0x12c06950
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 93 boot-framework.oat + 0x13297f5
18:31:45 INFO - sp = 0xffec19b0 pc = 0x721e27f7
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 94 system@[e-mail] + 0x378686
18:31:45 INFO - sp = 0xffec19b8 pc = 0x70090688
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 95 system@[e-mail] + 0x5d4f6
18:31:45 INFO - sp = 0xffec19d0 pc = 0x6fd754f8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 96 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec19d4 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 97 libart.so + 0x331a03
18:31:45 INFO - sp = 0xffec19d8 pc = 0xf33b0a05
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 98 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec19e0 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 99 boot-framework.vdex + 0x5db7f2
18:31:45 INFO - sp = 0xffec19e4 pc = 0xf18477f4
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 100 boot-framework.oat + 0x13297f5
18:31:45 INFO - sp = 0xffec1a18 pc = 0x721e27f7
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 101 boot-framework.oat + 0x1328de6
18:31:45 INFO - sp = 0xffec1a1c pc = 0x721e1de8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 102 dalvik-main space (region space) (deleted) + 0x5bce
18:31:45 INFO - sp = 0xffec1a2c pc = 0x12c05bd0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 103 dalvik-main space (region space) (deleted) + 0x73ae
18:31:45 INFO - sp = 0xffec1a30 pc = 0x12c073b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 104 system@[e-mail] + 0x378686
18:31:45 INFO - sp = 0xffec1a44 pc = 0x70090688
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 105 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec1a48 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 106 dalvik-main space (region space) (deleted) + 0x73ae
18:31:45 INFO - sp = 0xffec1a50 pc = 0x12c073b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 107 system@[e-mail] + 0x5d4f6
18:31:45 INFO - sp = 0xffec1a58 pc = 0x6fd754f8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 108 system@[e-mail] + 0x378686
18:31:45 INFO - sp = 0xffec1a5c pc = 0x70090688
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 109 libart.so + 0x332e5f
18:31:45 INFO - sp = 0xffec1a60 pc = 0xf33b1e61
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 110 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec1a64 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 111 libart.so + 0x3dba1f
18:31:45 INFO - sp = 0xffec1a70 pc = 0xf345aa21
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 112 boot-framework.vdex + 0x714e86
18:31:45 INFO - sp = 0xffec1ab8 pc = 0xf1980e88
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 113 dalvik-main space (region space) (deleted) + 0x5bce
18:31:45 INFO - sp = 0xffec1ac8 pc = 0x12c05bd0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 114 dalvik-zygote space (deleted) + 0x20c6
18:31:45 INFO - sp = 0xffec1acc pc = 0x726840c8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 115 dalvik-main space (region space) (deleted) + 0x4a46
18:31:45 INFO - sp = 0xffec1ad8 pc = 0x12c04a48
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 116 boot-framework.oat + 0x1325a13
18:31:45 INFO - sp = 0xffec1ae0 pc = 0x721dea15
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 117 system@[e-mail] + 0x2e1066
18:31:45 INFO - sp = 0xffec1ae4 pc = 0x6fff9068
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 118 dalvik-zygote space (deleted) + 0x20c6
18:31:45 INFO - sp = 0xffec1ae8 pc = 0x726840c8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 119 dalvik-main space (region space) (deleted) + 0x4a46
18:31:45 INFO - sp = 0xffec1aec pc = 0x12c04a48
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 120 dalvik-main space (region space) (deleted) + 0x4be6
18:31:45 INFO - sp = 0xffec1b00 pc = 0x12c04be8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 121 dalvik-main space (region space) (deleted) + 0x4bb6
18:31:45 INFO - sp = 0xffec1b04 pc = 0x12c04bb8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 122 dalvik-main space (region space) (deleted) + 0x72e6
18:31:45 INFO - sp = 0xffec1b14 pc = 0x12c072e8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 123 dalvik-main space (region space) (deleted) + 0x730e
18:31:45 INFO - sp = 0xffec1b18 pc = 0x12c07310
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 124 dalvik-main space (region space) (deleted) + 0x73ae
18:31:45 INFO - sp = 0xffec1b1c pc = 0x12c073b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 125 dalvik-main space (region space) (deleted) + 0x694e
18:31:45 INFO - sp = 0xffec1b20 pc = 0x12c06950
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 126 libart.so + 0x2e2093
18:31:45 INFO - sp = 0xffec1b30 pc = 0xf3361095
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 127 dalvik-main space (region space) (deleted) + 0x4bb6
18:31:45 INFO - sp = 0xffec1b38 pc = 0x12c04bb8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 128 dalvik-main space (region space) (deleted) + 0x73ae
18:31:45 INFO - sp = 0xffec1b4c pc = 0x12c073b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 129 boot.oat + 0x23adc7
18:31:45 INFO - sp = 0xffec1b50 pc = 0x70535dc9
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 130 system@[e-mail] + 0x159b02
18:31:45 INFO - sp = 0xffec1b54 pc = 0x6fb06b04
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 131 dalvik-main space (region space) (deleted) + 0x900086
18:31:45 INFO - sp = 0xffec1b60 pc = 0x13500088
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 132 dalvik-main space (region space) (deleted) + 0x9000ae
18:31:45 INFO - sp = 0xffec1b68 pc = 0x135000b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 133 dalvik-main space (region space) (deleted) + 0x45fe
18:31:45 INFO - sp = 0xffec1b70 pc = 0x12c04600
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 134 system@[e-mail] + 0xd8b6
18:31:45 INFO - sp = 0xffec1b74 pc = 0x6fb9f8b8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 135 dalvik-main space (region space) (deleted) + 0x900086
18:31:45 INFO - sp = 0xffec1bb8 pc = 0x13500088
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 136 dalvik-main space (region space) (deleted) + 0x901de6
18:31:45 INFO - sp = 0xffec1bbc pc = 0x13501de8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 137 dalvik-main space (region space) (deleted) + 0x9000ae
18:31:45 INFO - sp = 0xffec1bc0 pc = 0x135000b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 138 dalvik-main space (region space) (deleted) + 0x694e
18:31:45 INFO - sp = 0xffec1bc4 pc = 0x12c06950
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 139 boot-framework.oat + 0x13265d9
18:31:45 INFO - sp = 0xffec1bd0 pc = 0x721df5db
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 140 system@[e-mail] + 0x2600d6
18:31:45 INFO - sp = 0xffec1bd4 pc = 0x6ff780d8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 141 dalvik-main space (region space) (deleted) + 0x72e6
18:31:45 INFO - sp = 0xffec1bd8 pc = 0x12c072e8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 142 dalvik-main space (region space) (deleted) + 0x73ae
18:31:45 INFO - sp = 0xffec1be0 pc = 0x12c073b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 143 dalvik-main space (region space) (deleted) + 0x901de6
18:31:45 INFO - sp = 0xffec1be4 pc = 0x13501de8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 144 system@[e-mail] + 0x191a4e
18:31:45 INFO - sp = 0xffec1be8 pc = 0x6fea9a50
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 145 system@[e-mail] + 0x139ca6
18:31:45 INFO - sp = 0xffec1bec pc = 0x6fe51ca8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 146 boot-framework.oat + 0x132980f
18:31:45 INFO - sp = 0xffec1bf0 pc = 0x721e2811
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 147 system@[e-mail] + 0x38458e
18:31:45 INFO - sp = 0xffec1bf4 pc = 0x7009c590
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 148 dalvik-main space (region space) (deleted) + 0x901de6
18:31:45 INFO - sp = 0xffec1bf8 pc = 0x13501de8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 149 dalvik-zygote space (deleted) + 0x213e
18:31:45 INFO - sp = 0xffec1bfc pc = 0x72684140
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 150 system@[e-mail] + 0x139ca6
18:31:45 INFO - sp = 0xffec1c00 pc = 0x6fe51ca8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 151 system@[e-mail] + 0x191a4e
18:31:45 INFO - sp = 0xffec1c04 pc = 0x6fea9a50
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 152 system@[e-mail] + 0x19c3de
18:31:45 INFO - sp = 0xffec1c08 pc = 0x6feb43e0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 153 dalvik-zygote space (deleted) + 0x20e6
18:31:45 INFO - sp = 0xffec1c0c pc = 0x726840e8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 154 system@[e-mail] + 0x18cfa6
18:31:45 INFO - sp = 0xffec1c10 pc = 0x6fea4fa8
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 155 system@[e-mail] + 0x18d196
18:31:45 INFO - sp = 0xffec1c18 pc = 0x6fea5198
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 156 system@[e-mail] + 0x1acc4e
18:31:45 INFO - sp = 0xffec1c20 pc = 0x6fec4c50
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 157 system@[e-mail] + 0x17cd66
18:31:45 INFO - sp = 0xffec1c24 pc = 0x6fe94d68
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 158 system@[e-mail] + 0x18de
18:31:45 INFO - sp = 0xffec1c28 pc = 0x6fd198e0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 159 system@[e-mail] + 0x9fe
18:31:45 INFO - sp = 0xffec1c30 pc = 0x6fd18a00
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 160 boot-com.google.vr.platform.oat + 0x5ffe
18:31:45 INFO - sp = 0xffec1c34 pc = 0x72682000
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 161 dalvik-zygote space (deleted) + 0x213e
18:31:45 INFO - sp = 0xffec1c38 pc = 0x72684140
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 162 dalvik-main space (region space) (deleted) + 0x494e
18:31:45 INFO - sp = 0xffec1c3c pc = 0x12c04950
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 163 dalvik-zygote space (deleted) + 0x2166
18:31:45 INFO - sp = 0xffec1c4c pc = 0x72684168
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 164 system@[e-mail] + 0x3b1aaa
18:31:45 INFO - sp = 0xffec1c64 pc = 0x700c9aac
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 165 libart.so + 0x3dbd71
18:31:45 INFO - sp = 0xffec1c80 pc = 0xf345ad73
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 166 dalvik-zygote space (deleted) + 0x606
18:31:45 INFO - sp = 0xffec1c88 pc = 0x72682608
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 167 dalvik-main space (region space) (deleted) + 0x3375ef2b
18:31:45 INFO - sp = 0xffec1c90 pc = 0x4635ef2d
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 168 libart.so + 0x3e07ed
18:31:45 INFO - sp = 0xffec1cb8 pc = 0xf345f7ef
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 169 boot-framework.oat + 0x86826
18:31:45 INFO - sp = 0xffec1cd8 pc = 0x70f3f828
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 170 boot-framework.oat + 0x8682a
18:31:45 INFO - sp = 0xffec1cdc pc = 0x70f3f82c
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 171 dalvik-main space (region space) (deleted) + 0x4896
18:31:45 INFO - sp = 0xffec1d10 pc = 0x12c04898
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 172 system@[e-mail] + 0x9f9e
18:31:45 INFO - sp = 0xffec1d14 pc = 0x6fd21fa0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 173 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1d20 pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 174 system@[e-mail] + 0x38458e
18:31:45 INFO - sp = 0xffec1d24 pc = 0x7009c590
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 175 libart.so + 0xac881
18:31:45 INFO - sp = 0xffec1d40 pc = 0xf312b883
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 176 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1d48 pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 177 base.apk + 0x13f70be
18:31:45 INFO - sp = 0xffec1d68 pc = 0xbaba10c0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 178 system@[e-mail] + 0x38458e
18:31:45 INFO - sp = 0xffec1d70 pc = 0x7009c590
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 179 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1d84 pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 180 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1d8c pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 181 libart.so + 0x331a03
18:31:45 INFO - sp = 0xffec1d90 pc = 0xf33b0a05
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 182 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1d98 pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 183 libart.so + 0x358e2f
18:31:45 INFO - sp = 0xffec1dd0 pc = 0xf33d7e31
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 184 system@[e-mail] + 0x3b1c8e
18:31:45 INFO - sp = 0xffec1dd4 pc = 0x700c9c90
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 185 system@[e-mail] + 0x3b1c8a
18:31:45 INFO - sp = 0xffec1ddc pc = 0x700c9c8c
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 186 libart.so + 0xad085
18:31:45 INFO - sp = 0xffec1de0 pc = 0xf312c087
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 187 system@[e-mail] + 0x38458e
18:31:45 INFO - sp = 0xffec1df8 pc = 0x7009c590
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 188 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1e0c pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 189 libart.so + 0x33181f
18:31:45 INFO - sp = 0xffec1e18 pc = 0xf33b0821
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 190 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1e1c pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 191 boot-framework.vdex + 0xa410a2
18:31:45 INFO - sp = 0xffec1e34 pc = 0xf1cad0a4
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 192 boot-framework.vdex + 0xf9f24d
18:31:45 INFO - sp = 0xffec1e38 pc = 0xf220b24f
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 193 dalvik-main space (region space) (deleted) + 0x4896
18:31:45 INFO - sp = 0xffec1e48 pc = 0x12c04898
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 194 system@[e-mail] + 0x61176
18:31:45 INFO - sp = 0xffec1e4c pc = 0x6fd79178
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 195 libart.so + 0x2af151
18:31:45 INFO - sp = 0xffec1e60 pc = 0xf332e153
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 196 libandroid_runtime.so + 0x142a99
18:31:45 INFO - sp = 0xffec1e68 pc = 0xf3f7ba9b
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 197 libandroid_runtime.so + 0x142a94
18:31:45 INFO - sp = 0xffec1e6c pc = 0xf3f7ba96
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 198 system@[e-mail] + 0x61176
18:31:45 INFO - sp = 0xffec1e74 pc = 0x6fd79178
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 199 libart.so + 0x2904db
18:31:45 INFO - sp = 0xffec1e78 pc = 0xf330f4dd
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 200 system@[e-mail] + 0x38458e
18:31:45 INFO - sp = 0xffec1e90 pc = 0x7009c590
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 201 libart.so + 0x40688c
18:31:45 INFO - sp = 0xffec1e9c pc = 0xf348588e
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 202 libart.so + 0x43c4ae
18:31:45 INFO - sp = 0xffec1ea4 pc = 0xf34bb4b0
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 203 libart.so + 0x406884
18:31:45 INFO - sp = 0xffec1eac pc = 0xf3485886
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 204 libart.so + 0x28425f
18:31:45 INFO - sp = 0xffec1eb0 pc = 0xf3303261
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 205 libart.so + 0x40688c
18:31:45 INFO - sp = 0xffec1ebc pc = 0xf348588e
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 206 libart.so + 0x27a7a5
18:31:45 INFO - sp = 0xffec1ec8 pc = 0xf32f97a7
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 207 libart.so + 0x27a5f3
18:31:45 INFO - sp = 0xffec1efc pc = 0xf32f95f5
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 208 app_process32 + 0x4a99
18:31:45 INFO - sp = 0xffec1f04 pc = 0xaf1a5a9b
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 209 libandroid_runtime.so + 0x6f387
18:31:45 INFO - sp = 0xffec1f10 pc = 0xf3ea8389
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 210 libart.so + 0x27a5f3
18:31:45 INFO - sp = 0xffec1f14 pc = 0xf32f95f5
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 211 libart.so + 0x27a5f3
18:31:45 INFO - sp = 0xffec1f20 pc = 0xf32f95f5
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 212 libandroid_runtime.so + 0x70b97
18:31:45 INFO - sp = 0xffec1f24 pc = 0xf3ea9b99
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 213 dalvik-main space (region space) (deleted) + 0x1f3cefa1
18:31:45 INFO - sp = 0xffec1f3c pc = 0x31fcefa3
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 214 libart.so + 0xb1951
18:31:45 INFO - sp = 0xffec1f40 pc = 0xf3130953
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 215 libart.so + 0x2330d7
18:31:45 INFO - sp = 0xffec1f44 pc = 0xf32b20d9
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 216 libart.so + 0x23345b
18:31:45 INFO - sp = 0xffec1f48 pc = 0xf32b245d
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 217 libc.so + 0x6a4b3
18:31:45 INFO - sp = 0xffec1f58 pc = 0xf4c7e4b5
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 218 app_process32 + 0x1b23
18:31:45 INFO - sp = 0xffec1fc8 pc = 0xaf1a2b25
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 219 app_process32 + 0x4e27
18:31:45 INFO - sp = 0xffec1fd0 pc = 0xaf1a5e29
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 220 libandroid_runtime.so + 0x1631d2
18:31:45 INFO - sp = 0xffec2018 pc = 0xf3f9c1d4
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 221 dalvik-main space (region space) (deleted) + 0x1aa9625f
18:31:45 INFO - sp = 0xffec2058 pc = 0x2d696261
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 222 dalvik-main space (region space) (deleted) + 0x19a13774
18:31:45 INFO - sp = 0xffec205c pc = 0x2c613776
18:31:45 INFO - Found by: stack scanning
18:31:45 INFO - 223 dalvik-main space (region space) (deleted) + 0x20b37363
18:31:45 INFO - sp = 0xffec2080 pc = 0x33737365
18:31:45 INFO - Found by: stack scanning

Comment 9

[Intermittent Failures Robot]

914 failures in 2561 pushes (0.357 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 616
• mozilla-inbound: 182
• try: 25
• mozilla-central: 89
• mozilla-beta: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 366
• android-hw-p2-8-0-arm7-api-16-pgo: 541
• android-em-7-0-x86_64: 6
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

:aryx, can we disable these tests on inbound/autoland until they are fixed or consider backing out bug 1335740

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1335740 has been backed out from autoland.

Comment 12

[Intermittent Failures Robot]

154 failures in 433 pushes (0.356 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 92
• mozilla-inbound: 53
• mozilla-central: 7
• try: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 56
• android-hw-p2-8-0-arm7-api-16-pgo: 92
• android-em-7-0-x86_64: 3
• android-em-4-3-armv7-api16-pgo: 2
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-25&endday=2019-03-25&tree=all

Comment 13

[Intermittent Failures Robot]

179 failures in 3910 pushes (0.046 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 102
• mozilla-inbound: 55
• mozilla-central: 8
• try: 13
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 63
• android-em-7-0-x86_64: 12
• android-hw-p2-8-0-arm7-api-16-pgo: 101
• android-em-4-3-armv7-api16-pgo: 2
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 14

[Intermittent Failures Robot]

10 failures in 2843 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 1
• mozilla-inbound: 2
• autoland: 7

Platform breakdown:

• android-em-7-0-x86_64: 4
• android-hw-p2-8-0-arm7-api-16: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 15

[Intermittent Failures Robot]

13 failures in 3025 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-inbound: 2
• mozilla-beta: 2
• autoland: 7
• mozilla-central: 2

Platform breakdown:

• android-em-7-0-x86_64: 10
• android-hw-p2-8-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 16

[Intermittent Failures Robot]

14 failures in 2673 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-inbound: 3
• mozilla-central: 2
• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 10
• android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 17

[Intermittent Failures Robot]

4 failures in 3031 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 1
• android-hw-p2-8-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 18

[Geoff Brown [:gbrown]]

This bug is often being confused with bug 1534732, for android-em-7-0-x86_64.

From what I can see, the android-hw failures are completely different from the android-em ones.

Comment 19

[Geoff Brown [:gbrown]]

The recent android-hw mochitest-media failures seem to go silent shortly after reporting something like:

01:21:56     INFO -  04-25 18:15:03.093 E/GeckoConsole( 6222): [JavaScript Error: "TypeError: aTab is null" {file: "chrome://browser/content/browser.js" line: 1462}]
01:21:56     INFO -  04-25 18:15:03.093 E/GeckoConsole( 6222): _handleTabSelected@chrome://browser/content/browser.js:1462:5
01:21:56     INFO -  04-25 18:15:03.093 E/GeckoConsole( 6222): onEvent@chrome://browser/content/browser.js:1995:14
01:21:56     INFO -  04-25 18:15:03.239 D/GeckoDistribution( 6222): Custom distribution directory not found.
01:21:56     INFO -  04-25 18:15:03.266 W/GeckoEventDispatcher( 6222): No listener for FormHistory:Ready
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): [JavaScript Error: "nativeTab is null" {file: "chrome://geckoview/content/ext-utils.js" line: 741}]
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): wrapTab@chrome://geckoview/content/ext-utils.js:741:5
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): TabManagerBase/this._tabs<@chrome://extensions/content/parent/ext-tabs-base.js:1791:49
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): get@resource://gre/modules/ExtensionUtils.jsm:69:20
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): getWrapper@chrome://extensions/content/parent/ext-tabs-base.js:1871:25
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): get activeTab@chrome://geckoview/content/ext-utils.js:704:23
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): candidates@chrome://extensions/content/parent/ext-tabs-base.js:1928:11
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): query@chrome://extensions/content/parent/ext-tabs-base.js:1947:7
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): query@chrome://geckoview/content/ext-tabs.js:384:24
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): call/result</<@resource://gre/modules/ExtensionParent.jsm:968:49
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): withPendingBrowser@resource://gre/modules/ExtensionParent.jsm:622:26
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): call/result<@resource://gre/modules/ExtensionParent.jsm:967:24
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): withTiming@resource://gre/modules/ExtensionParent.jsm:934:14
01:21:56     INFO -  04-25 18:15:03.275 E/GeckoConsole( 6222): call@resource://gre/modules/ExtensionParent.jsm:966:25
01:21:56     INFO -  04-25 18:15:03.356 W/GeckoEventDispatcher( 6222): No listener for GeckoView:AccessibilityReady
01:21:56     INFO -  04-25 18:15:03.383 E/GeckoConsole( 6222): [JavaScript Error: "Unexpected content-type "text/html;charset=utf-8"" {file: "resource://services-settings/Utils.jsm" line: 87}]
01:21:56     INFO -  04-25 18:15:03.383 E/GeckoConsole( 6222): fetchLatestChanges@resource://services-settings/Utils.jsm:87:15

Comment 20

[Andreea Pavel [:apavel]]

(In reply to Geoff Brown [:gbrown] from comment #18)

This bug is often being confused with bug 1534732, for android-em-7-0-x86_64.

From what I can see, the android-hw failures are completely different from the android-em ones.

That's probably because if the bug is not suggested in TH we search for remoteautomation.py | application timed out thus the correct bug is not showed in the search results.

Comment 21

[Intermittent Failures Robot]

1 failures in 4131 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 22

[Intermittent Failures Robot]

1 failures in 4705 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 23

[Intermittent Failures Robot]

1 failures in 4658 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 24

[Intermittent Failures Robot]

2 failures in 4730 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 25

[Intermittent Failures Robot]

4 failures in 4337 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 2
• autoland: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 26

[Bob Clary [:bc] (inactive)]

snorp: If you follow the orangefactor link and then look at the logs for the failures in the last week (there are only 4) you can see these are all "hangs" at start up. Note the time in minutes:seconds. Then follow the link to the logcat and scroll to the relevant section corresponding to the minutes:seconds from the original log, you'll find the gecko log messages from startup.

Can you take a look and see if there is something obvious to you about why fennec just randomly hangs on startup?

Comment 27

[Geoff Brown [:gbrown]]

I think comment 19 remains valid, which means fennec is blowing up at

https://searchfox.org/mozilla-central/rev/f8b11433159cbc9cc80500b3e579d767473fa539/mobile/android/chrome/content/browser.js#1480

because getTabForId() is returning null at

https://searchfox.org/mozilla-central/rev/f8b11433159cbc9cc80500b3e579d767473fa539/mobile/android/chrome/content/browser.js#2007.

(I don't know how that situation would arise.)

Comment 28

[Intermittent Failures Robot]

2 failures in 5197 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 4753 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 30

[Intermittent Failures Robot]

2 failures in 4409 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 31

[Intermittent Failures Robot]

15 failures in 602 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 15

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-02&endday=2019-07-02&tree=all

Comment 32

[Intermittent Failures Robot]

27 failures in 175 pushes (0.154 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 27

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 22
• android-em-7-0-x86_64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-06&endday=2019-07-06&tree=all

Comment 33

[Intermittent Failures Robot]

53 failures in 3501 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-inbound: 25
• try: 27
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 48
• android-em-7-0-x86_64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 34

[Intermittent Failures Robot]

43 failures in 551 pushes (0.078 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 16
• mozilla-beta: 27

Platform breakdown:

• android-em-7-0-x86_64: 22
• android-hw-p2-8-0-arm7-api-16: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-08&endday=2019-07-08&tree=all

Comment 35

[Andreea Pavel [:apavel]]

Recent occurrences here are missclasifications and are actually Bug 1563997.

Comment 36

[James Willcox (:snorp) ([e-mail]) (he/him)]

I don't see anything obvious here. Maybe some kind of race in Fennec?

Comment 37

[Intermittent Failures Robot]

15 failures in 893 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 10
• try: 5

Platform breakdown:

• android-em-7-0-x86_64: 11
• android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-11&endday=2019-07-11&tree=all

Comment 38

[Andreea Pavel [:apavel]]

James, recent classifications here are Bug 1563997, the failure rate on this bug is close to none.
There is just one failure from the 7th https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255156334&repo=mozilla-central&lineNumber=2994

Comment 39

[Intermittent Failures Robot]

33 failures in 4710 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 21
• mozilla-beta: 12

Platform breakdown:

• android-em-7-0-x86_64: 19
• android-hw-p2-8-0-arm7-api-16: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 40

[Intermittent Failures Robot]

16 failures in 792 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 16

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-15&endday=2019-07-15&tree=all

Comment 41

[Intermittent Failures Robot]

16 failures in 4298 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 16

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 42

[Bob Clary [:bc] (inactive)]

I see tombstones with these on beta and it is weird these are fennec_aurora's for beta I would think. Confirmed the apk is for fennec_aurora.

signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Abort message: 'PRODUCTION: Adjust is running in Production mode. Use this setting only for the build that you want to publish. Set the environment to sandbox if you want to test your app!'

Comment 43

[James Willcox (:snorp) ([e-mail]) (he/him)]

Some sort of misconfiguration with Adjust? I don't really know anything about that. -> stefan

Comment 44

[Intermittent Failures Robot]

2 failures in 4350 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1537745&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 45

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Unclear if there is anything actionable here. Please reopen if this is still important.