1150924 - [Crash] [@ jemalloc_crash | arena_dalloc | je_realloc | replace_realloc ]

Status: RESOLVED WONTFIX

(Core :: DOM: Core & HTML, defect)

Description

[Nicholas Troast [:ntroast]]

We observed the following crash signature during testing.

[@ jemalloc_crash | arena_dalloc | je_realloc | replace_realloc ]

Cafbot will upload the decoded minidump and extra file.

Comment 1

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment -

Comment 2

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump -

Comment 3

[Mike Lee [:mlee]]

Johnny & Andrew,

Can you have someone(s) on the DOM team familiar with the media codebase look into this critical fxOS 2.2 crash asap. It's currently blocking CAF and is directly contributing to their inability to reach our MTBF goal.

As much as possible we need these fixed and available for CAF testing by Sunday, April 5th.

Thanks,
Mike

Comment 4

[Nicholas Troast [:ntroast]]

Hi Mike, I think you may be confusing this bug with another.
This is [@ jemalloc_crash | arena_dalloc | je_realloc | replace_realloc ]

Comment 5

[Mike Lee [:mlee]]

Hi Nick,

I added that signature after looking into the decoded minidump. Although jemalloc_crash is the last signature that looks to be code that could be entered from various paths. I added the other signature [1] since it appears to be the latest least-generic signature.

Mike

[1] mozilla::MediaSegmentBase<mozilla::AudioSegment, mozilla::AudioChunk>::AppendSlice

Comment 6

[Nicholas Troast [:ntroast]]

Oh! Sorry about that then. I seem to be the confused one :)

Comment 7

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119
Moz BuildID: 20150330002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=473cd63f53c855299b719285d9b95e3f2910782f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e44e28d2a37258fe0c908c59f2e91e5287777b40
Patches: bug 1133398, bug 1143694, bug 1146987, bug 1145724, bug 1147646, bug 1133147, bug 1150271, bug 1142770

Comment 8

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 9

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 10

[Bobby Holley (:bholley)]

The crash is in MediaStream code, which I'm not familiar with - it's used mostly by webrtc and webaudio. I'm not the right person to fix this, but here's my cursory analysis as a general Gecko developer:

The particular crash is here:

nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::EnsureCapacity

Which means that this is an OOM. The OOM happens here:

mozilla::MediaSegmentBase<mozilla::AudioSegment, mozilla::AudioChunk>::AppendSlice

Which is basically appending a mozilla::AudioChunk to an nsTArray. AudioChunk is defined in AudioSegment.h, but it doesn't appear to be very large. It holds onto potentially-large data via pointers, so that shouldn't be an issue when appending one of these things to the nsTArray.

It's worth verifying that our memory is basically shot by the time we hit this. If that's the case, then something else is chewing up memory and this is just how we die. If not, then I don't understand how we'd be dying in infallible allocation here.

Comment 11

[Bobby Holley (:bholley)]

Next steps would be to get people familiar with device debugging and OOM failures to have a look to see if we're OOMing and figure out if anything in particular is leaking or hogging memory.

Comment 12

[Ting-Yu Chou [:ting] (away)]

From attachment 8588008 [details], the crash is because the arena going to be freed has corrupted magic signature, which I think STR is required to proceed.

Comment 13

[Inder]

Ting, this crash was produced during stability tests which involves monkey testing for several hours and there is no clear STR for this. If we are not able to identify the issue using provided logs then please feel free to provide us a debug patch with additional logging to identify the issue.

Comment 14

[Bobby Holley (:bholley)]

(In reply to Ting-Yu Chou [:ting] from comment #12)
> From attachment 8588008 [details], the crash is because the arena going to
> be freed has corrupted magic signature, which I think STR is required to
> proceed.

You're right - I should have inspected more closely. This is corruption, not OOM:

https://hg.mozilla.org/mozilla-central/annotate/eeb9438975a5/memory/mozjemalloc/jemalloc.c#l4711

(In reply to Inder from comment #13)
> Ting, this crash was produced during stability tests which involves monkey
> testing for several hours and there is no clear STR for this. If we are not
> able to identify the issue using provided logs then please feel free to
> provide us a debug patch with additional logging to identify the issue.

Then this is probably going to be very hard to make progress on. Somebody somewhere in Gecko is corrupting memory somewhere. That's not really a lot to go on.

Comment 15

[Kai-Zhen Li [:seinlin][:kli]]

Inder, could you try to disable mozjemalloc as bug 1148324 comment 9 and use the one in bionic?

Comment 16

[Ting-Yu Chou [:ting] (away)]

(In reply to Inder from comment #13)
> Ting, this crash was produced during stability tests which involves monkey
> testing for several hours and there is no clear STR for this. If we are not
> able to identify the issue using provided logs then please feel free to
> provide us a debug patch with additional logging to identify the issue.

It's really hard to debug memory corruption by adding logs. I'd like to start from reproducing this locally. Was it crashed while running monkey test or some other stability tests?

BTW, attachment 8588007 [details] has following line before crash, anyone knows how to trigger this behavior?

04-02 17:49:25.231   260   752 D audio_hw_primary: start_output_stream: enter: stream(0xb8a4b170)usecase(1: low-latency-playback) devices(0x2)

Comment 17

[Ting-Yu Chou [:ting] (away)]

Attachment: debug patch

A debug patch which prints out 8 bytes before/after corrupted magic, see if it could provide any clues.

Comment 18

[Inder]

Nick - please land the patch in our build.

Comment 19

[Nicholas Troast [:ntroast]]

The patch was added, but it may or may not be built into the very next AU.

Comment 20

[shawn ku [:sku]]

Seinin, Please help update this bug.

Comment 21

[Bobby Chien]

according to bug 1151787, this error should not appear again.