1180954 - Crash in JS:CollectRuntimeStats while stability testing

Status: RESOLVED DUPLICATE of bug 1132502

(Core :: JavaScript Engine, defect, P2)

Description

[Greg Grisco]

Saw this stack trace while running stability tests overnight:

@ AddClassInfo | StatsCellCallback<(Granularity)0u> | IterateCompartmentsArenasCells | js::IterateZonesCompartmentsArenasCells ]

cafbot will upload the logs.

Comment 1

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment -

Comment 2

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump -

Comment 3

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.196
Moz BuildID: 20150606002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.196.xml?h=release
B2G Version: v2.2
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=8fc797527a3eca7665bc1d1828848f2fb77ca99f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e0045f9c8b7e84fc52ba628141688c8ecb4b7a52
Patches: bug 1133147, bug 1167799, bug 1162663

Comment 4

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.196

Comment 5

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.196

Comment 6

[Jon Coppeard (:jonco)]

Looks like a use after free accessing the contents of a BaseShape's JSClass.

Comment 7

[Hema Koka [:hema]]

Naveed/Jon, 

Your help is appreciated with this stability bug. If you need further info, feel free to ping ggrisco or ikumar from CAF

(Also NI Josh Cheng - 2.2 RM to triage and track 2.2 crash issues)

Thanks
Hema

Comment 8

[Jon Coppeard (:jonco)]

It seems there have been no dynamically allocated JSClasses since FF31 (bug 990290), so not UAF of a JSClass.

Does this happen every time?  Is it possible to reproduce?

Comment 9

[Naveed Ihsanullah [:naveed]]

jonco is active here so pulling off the needinfo on me

Comment 10

[Greg Grisco]

(In reply to Jon Coppeard (:jonco) from comment #8)
> It seems there have been no dynamically allocated JSClasses since FF31 (bug
> 990290), so not UAF of a JSClass.
> 
> Does this happen every time?  Is it possible to reproduce?

So far we only saw this crash one time on AU 196 after many hours of test, so it's not easily reproduced.

Comment 11

[Josh Cheng [:josh]]

Dear Jon,
Thanks for your help.
Is it possible to find any clue from the log provided earlier?

Comment 12

[Jon Coppeard (:jonco)]

It's not a lot to go on.  I'm wondering if something in the memory reporter can end up triggering a GC while we're iterating through the arenas, but I'd be surprised if that didn't trigger an assert somewhere.

Comment 13

[Josh Cheng [:josh]]

(In reply to Jon Coppeard (:jonco) (PTO until 21st July) from comment #12)
> It's not a lot to go on.  I'm wondering if something in the memory reporter
> can end up triggering a GC while we're iterating through the arenas, but I'd
> be surprised if that didn't trigger an assert somewhere.

Thanks Jon,
It seems we can only wait until same issue happen next time? Before then, is there any additional log we can ask Greg to provide?

Comment 14

[cafbot (PoC: ggrisco)]

"Closing issue which has not been seen since 06/30/15 20:21"

Comment 15

[Greg Grisco]

Re-opening since this was seen again on AU 214.  cafbot will follow-up with latest logs.

Comment 16

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment -

Comment 17

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump -

Comment 18

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.214
Moz BuildID: 20150606002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.214.xml?h=release
B2G Version: v2.2
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=8fc797527a3eca7665bc1d1828848f2fb77ca99f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e0045f9c8b7e84fc52ba628141688c8ecb4b7a52
Patches: bug 1133147, bug 1181641

Comment 19

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.214

Comment 20

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.214

Comment 21

[Josh Cheng [:josh]]

Hi Jon,
Could you help to check whether there are any useful information in new log?
Thanks!

Comment 22

[Jon Coppeard (:jonco)]

It's the same problem as before, crashing when trying to dereferencing the heap free pattern while inside the memory reporter.

Nick, have you seen anything like this before?

Comment 23

[Nicholas Nethercote [inactive]]

The JS memory reporter iterates over every live thing in the GC heap, and measures most of the malloc'd blocks that hang off those things. So if there's any kind of GC heap corruption there's a good chance that it will manifest in the reporter.

This reminds me slightly of part 6 in bug 972712 which involved identifying which class each object belonged to. It originally landed in March 2014 but had to be backed out due to intermittent ASAN failures. I was eventually able to land it three months later (in bug 1023719) despite not having made any changes. I concluded that I had been hitting some kind of latent heap corruption and then it went away, either via luck or via someone fixing something.

So this one is going to be difficult to debug. I wonder if implementing a GC heap sanity checker would be a good idea. It would be similar to the reporter -- iterate over every GC thing, checking that everything looks ok. I've seen this kind of thing be implemented in other systems, mostly checking IR in compilers, and they typically find real problems.

Comment 24

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.221
Moz BuildID: 20150721162504
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.221.xml?h=release
B2G Version: v2.2
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=e1e6317f17a840b19af9dbb25f5a771d8d9fa161
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=72899a6655a5d42cea9bb5c3effe1d720bd8bb4e
Patches: bug

Comment 25

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.221

Comment 26

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.221

Comment 27

[Inder]

Please feel free to provide us a debug patch to collect more logs. We are seeing this issue more consistently now after many hours of stability testing so we need to resolve it asap.

Comment 28

[Josh Cheng [:josh]]

Hi Nick,
Is it possible to provide a debug patch here? Thanks!

Comment 29

[Nicholas Nethercote [inactive]]

> Is it possible to provide a debug patch here? Thanks!

I don't have anything specific for you, sorry.

Comment 30

[Josh Cheng [:josh]]

Hi Bobby,
Could you help to find anyone who can help here?
Thanks!

Comment 31

[Jon Coppeard (:jonco)]

This looks like the same crash reported in bug 1189934.

Comment 32

[Bobby Chien]

Similar crash from memory reference in bug 1189934 and bug 1132502.

Comment 33

[Bobby Chien]

Jason, per some research in comment 32. Looks similar crash in Javascript. could you help on this?

Comment 34

[Bobby Chien]

Naveed, could you help this bug? Thanks.

Comment 35

[Naveed Ihsanullah [:naveed]]

Terrence check out Nick's comment, https://bugzilla.mozilla.org/show_bug.cgi?id=1180954#c23, would that help us out in the future?

Comment 37

[Yorgos]

Attachment: memory-report.json.gz

on a OS capable up to 3,3 Gb ram.

Comment 38

[Bobby Chien]

Yorgos, you probably pasted wrong crash link in comment 36 and comment 37. could you clarify this? Thanks.

Comment 39

[Bobby Chien]

I got it, Yorgos. Sorry for confusion.

Comment 40

[Bobby Chien]

Revisit minidump attachments and search signature from crash report site. No new similar crash for long time. Close as worksforme.

Comment 42

[Yorgos]

@Bobby: I dont understand your request. 
Minidum folder: empty.
I found only
https://crash-stats.mozilla.com/report/index/6aaae493-b969-41da-ae75-badf82151102
where sould I search for the signature?

Comment 43

[Yorgos]

http://postimg.org/image/s6e21f9lf/

Comment 44

[Bobby Chien]

(In reply to Yorgos from comment #42)
> @Bobby: I dont understand your request. 
> Minidum folder: empty.
> I found only
> https://crash-stats.mozilla.com/report/index/6aaae493-b969-41da-ae75-
> badf82151102
> where sould I search for the signature?

Per your crash (https://crash-stats.mozilla.com/report/index/6aaae493-b969-41da-ae75-badf82151102), you could copy signature "js::detail::HashTable<T>::lookupForAdd" to search (top-right corner) from crash report site. so you could query statistics. Let me know if you still have trouble. Thanks.

Comment 45

[Yorgos]

Done.
http://postimg.org/image/sgc0wl7yb/
I am not the only with this crash, I hope for a fix.

Comment 46

[Bobby Chien]

Hi Jorgos, I close this bug for firefox OS. I saw you follow another bug 1132502 for firefox. Please keep trace. Many thanks.

Comment 47

[Nicholas Nethercote [inactive]]

Yeah, turns out this is actually a dup of bug 1132502.