1593980 - Intermittent [tier2] layout/base/crashtests/1586600.html | application crashed [@ mozalloc_handle_oom(unsigned int)]

Status: RESOLVED DUPLICATE of bug 1594446

(Core :: Graphics: WebRender, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: opoprus [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=274576462&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/Lm87T1Y8SGGym0L39Z_CAg/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Lm87T1Y8SGGym0L39Z_CAg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2019-11-05T09:55:52.579Z] 09:55:20 INFO - REFTEST TEST-START | http://10.7.205.209:8854/tests/layout/base/crashtests/1586600.html
[task 2019-11-05T09:55:52.579Z] 09:55:20 INFO - REFTEST TEST-LOAD | http://10.7.205.209:8854/tests/layout/base/crashtests/1586600.html | 1940 / 3782 (51%)
[task 2019-11-05T09:55:52.579Z] 09:55:45 INFO - wait for org.mozilla.geckoview.test complete; top activity=com.bitbar.testdroid.monitor
[task 2019-11-05T09:55:52.579Z] 09:55:45 INFO - remoteautomation.py | Application ran for: 0:05:51.155797
[task 2019-11-05T09:55:52.579Z] 09:55:46 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/IiFuBefBS7G-Ekfd4vsuoA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-11-05T09:55:52.579Z] 09:55:48 INFO - REFTEST INFO | Copy/paste: /builds/task_1572947197/workspace/build/linux64-minidump_stackwalk /tmp/tmpAaXaUZ/07c6edc4-15d3-18d0-4a34-9d1ea6cd591c.dmp /tmp/tmpkK7bDg
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - REFTEST INFO | Saved minidump as /builds/task_1572947197/workspace/build/blobber_upload_dir/07c6edc4-15d3-18d0-4a34-9d1ea6cd591c.dmp
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - REFTEST INFO | Saved app info as /builds/task_1572947197/workspace/build/blobber_upload_dir/07c6edc4-15d3-18d0-4a34-9d1ea6cd591c.extra
[task 2019-11-05T09:55:52.579Z] 09:55:52 WARNING - REFTEST PROCESS-CRASH | http://10.7.205.209:8854/tests/layout/base/crashtests/1586600.html | application crashed [@ mozalloc_handle_oom(unsigned int)]
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - Crash dump filename: /tmp/tmpAaXaUZ/07c6edc4-15d3-18d0-4a34-9d1ea6cd591c.dmp
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - Operating system: Android
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - 0.0.0 Linux 4.4.56-g594d847d09a1 #1 SMP PREEMPT Thu Oct 26 22:34:08 UTC 2017 armv8l
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - CPU: arm
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - ARMv1 Qualcomm part(0x51008010) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - 8 CPUs
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - GPU: UNKNOWN
[task 2019-11-05T09:55:52.579Z] 09:55:52 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Crash address: 0x0
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Process uptime: not available
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Thread 54 (crashed)
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 0 libmozglue.so!mozalloc_abort [mozalloc_abort.cpp:95dfa337c1dbb77e210abc30790ccb49ff7e163b : 33 + 0x10]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r0 = 0x00000000 r1 = 0x00000021 r2 = 0x00430000 r3 = 0xe65b43bc
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0xb447e458 r5 = 0xb447e458 r6 = 0x00000004 r7 = 0xb447e450
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 r12 = 0xe9767d5c
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - fp = 0x00000001 sp = 0xb447e448 lr = 0xe65b405d pc = 0xcb24ff44
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: given as instruction pointer in context
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 1 libmozglue.so!mozalloc_handle_oom(unsigned int) [mozalloc_oom.cpp:95dfa337c1dbb77e210abc30790ccb49ff7e163b : 51 + 0x3]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0x00000001 r5 = 0xb447e458 r6 = 0x00000004 r7 = 0xb447e498
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e458 lr = 0xcb2500c1 pc = 0xcb2500c1
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 2 libxul.so!gkrust_shared::oom_hook::hook [lib.rs:95dfa337c1dbb77e210abc30790ccb49ff7e163b : 271 + 0x3]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0x77c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb447e4a0
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e4a0 lr = 0xc8fda3e1 pc = 0xc8fda3e1
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 3 libxul.so!rust_oom [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 216 + 0x1]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0x77c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb447e5c8
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e4a8 pc = 0xc95c2259
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 4 libxul.so!alloc::alloc::handle_alloc_error [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 248 + 0x3]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0x77c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb447e5c8
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e4b0 pc = 0xc95c7e57
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 5 libxul.so!<webrender_bindings::moz2d_renderer::Moz2dBlobRasterizer as webrender_api::image::AsyncBlobImageRasterizer>::rasterize [moz2d_renderer.rs:95dfa337c1dbb77e210abc30790ccb49ff7e163b : 528 + 0x1b]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0x77c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb447e5c8
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xb447e5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e4b8 pc = 0xc9138497
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 6 libxul.so!webrender::scene_builder_thread::Transaction::rasterize_blobs [scene_builder_thread.rs:95dfa337c1dbb77e210abc30790ccb49ff7e163b : 73 + 0x3]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0xa5dfd6d0 r5 = 0xc91380e1 r6 = 0x00000001 r7 = 0xb447e610
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xa5dfd6d0 r9 = 0xb1b7541c r10 = 0x00000001 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e5d0 pc = 0xc9192cdd
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - 7 libxul.so!std::sys_common::backtrace::__rust_begin_short_backtrace [backtrace.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 77 + 0xab]
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r4 = 0xb447e680 r5 = 0xb447e730 r6 = 0xb447e7e0 r7 = 0xb447e8b0
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - r8 = 0xa5dfd6d0 r9 = 0xb1b7541c r10 = 0x00000001 fp = 0x00000001
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - sp = 0xb447e618 pc = 0xc92820d9
[task 2019-11-05T09:55:52.584Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 8 libxul.so!core::ops::function::FnOnce::call_once{{vtable.shim}} [function.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 235 + 0x57]
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r4 = 0xb447e900 r5 = 0xb447e8c0 r6 = 0x00000000 r7 = 0xb447e8f8
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r8 = 0xb48295e0 r9 = 0xca27dff0 r10 = 0xc95c2e25 fp = 0x00000002
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e8b8 pc = 0xc9281fb3
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 9 libxul.so!<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once [boxed.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 787 + 0x29]
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r4 = 0x0000002c r5 = 0xb447e900 r6 = 0xb447e930 r7 = 0xb447e940
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r8 = 0xb48295e0 r9 = 0xca27dff0 r10 = 0xc95c2e25 fp = 0x00000002
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e900 pc = 0xc95c163d
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 10 libxul.so!std::sys::unix::thread::Thread::new::thread_start [thread.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 79 + 0x7]
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r4 = 0xb6aef9d0 r5 = 0xb447e970 r6 = 0xb447e970 r7 = 0x00000078
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r8 = 0x00001831 r9 = 0x0000186f r10 = 0xc95c2e25 fp = 0x00000002
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e950 pc = 0xc95c2e31
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 11 libc.so + 0x47947
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r4 = 0xb447e970 r5 = 0xb447e970 r6 = 0xb447e970 r7 = 0x00000078
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - r8 = 0x00001831 r9 = 0x0000186f r10 = 0xc95c2e25 fp = 0x00000002
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e958 pc = 0xe971d949
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: call frame info
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 12 libc.so + 0x4792f
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e95c pc = 0xe971d931
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: stack scanning
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 13 libc.so + 0x1b381
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e960 pc = 0xe96f1383
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: stack scanning
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 14 libc.so + 0x4792f
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e968 pc = 0xe971d931
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: stack scanning
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - 15 libxul.so!std::sys::unix::thread::Thread::new + 0x1b3
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - sp = 0xb447e9a4 pc = 0xc95c2e25
[task 2019-11-05T09:55:52.592Z] 09:55:52 INFO - Found by: stack scanning

Comment 1

[Eric Rahm [:erahm]]

This appears to be an OOM in WebRender.

Comment 2

[Intermittent Failures Robot]

29 failures in 1017 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 6

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-05&endday=2019-11-05&tree=all

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

It looks like we are hitting < 1% of memory free right from the start of the crashtest. I don't see any process dump in the resource usage json to explore what is holding onto all that memory, as it may not be us if it starts bad.

Comment 4

[Intermittent Failures Robot]

19 failures in 864 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-06&endday=2019-11-06&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1593980 - Disable layout/base/crashtests/1586600.html on Android 8.0 because it permafails. r?gbrown

Comment 6

[Intermittent Failures Robot]

69 failures in 4850 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #22 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 52
• mozilla-central: 15
• mozilla-inbound: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 69

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 7

[Intermittent Failures Robot]

28 failures in 4207 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 22
• mozilla-inbound: 2
• mozilla-central: 3
• try: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 8

[Andreea Pavel [:apavel]]

This had a spike between Nov 4th and 17th.

Aryx is the disable here still needed? If so, can you add the checking tag?

Comment 9

[Jamie Nicol [:jnicol]]

I don't think the disable will help: there are quite a few bugs recently for OOM during the crashtests on pixel2+webrender. Different tests during which we crash, or different functions in which we crash, but presumably just the same root cause. Disabling this test might just cause another to increase.

Comment 10

[Andreea Pavel [:apavel]]

Thank you.

Comment 11

[Intermittent Failures Robot]

9 failures in 5444 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 8
• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:jbonisteel, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 13

[Intermittent Failures Robot]

1 failures in 4331 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 5144 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593980&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 15

[Jamie Nicol [:jnicol]]

No failures since Dec 6th, Fixed by bug 1594446