1017740 - Crash in js::NewObjectWithClassProtoCommon() while running stability scripts

Status: RESOLVED DUPLICATE of bug 964537

(Core :: JavaScript Engine, defect)

Description

[Greg Grisco]

Attachment: decoded minidump of crash

Crash in stability test while running multimedia and scripts

[@ ? | js::NewObjectWithClassProtoCommon(js::ExclusiveContext*, js::Class const*, JSObject*, JSObject*, js::gc::AllocKind, js::NewObjectKind) ]

I don't have better STR unfortunately.  We have seen 3 times so far in stability test where we run these scripts over the weekend.

Not sure if this is related to bug 1016512.

Comment 1

[Greg Grisco]

Attachment: EXTRA file attachment (plus some logs)

Comment 2

[cafbot (PoC: ggrisco)]

Crash observed on: 

Device: msm8610
Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.108
Moz BuildID: 20140521000202
B2G Version: 1.4
Gecko Version: 30.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=93623f6435849cc9f54d9996e8e64828ac9091d1
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e15e80174566f3267b045e1bc95d8d0f6fa76780

Comment 3

[cafbot (PoC: ggrisco)]

Crash observed on: 

Device: msm8610
Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.112
Moz BuildID: 20140524000202
B2G Version: 1.4
Gecko Version: 30.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=8f4201a44676eb70926a3d2645d94bf92fcd6718
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=c69048b9a3fcacc456f281db08a5e6162655ecec

Comment 4

[cafbot (PoC: ggrisco)]

Crash observed on: 

Device: msm8226
Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.114
Moz BuildID: 20140528000201
B2G Version: 1.4
Gecko Version: 30.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=cd595be0a8e975559e8938830df5face89bec3e8
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=d591b0c691da6847dcb9a4f626211b597e8807fe

Comment 5

[cafbot (PoC: ggrisco)]

Crash observed on: 

Device: msm8610
Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.121
Moz BuildID: 20140604000202
B2G Version: 1.4
Gecko Version: 30.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=0c16adced7c51f795ef250aebe184f60b6a9b987
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=157a45f1fa280296dc9204de6def0b5b370ed2bd

Comment 6

[Nicolas B. Pierron [:nbp]]

As Bug 1017757, this stack is not actionable and we are not sure if it is correct.
The only thing that tell us is that we are in compiled code.

Greg, would it be possible to have some human-readable interpretation of the RIL communication (from the logcat) which is going on, such as we can figure out what are the last pieces of JavaScript which were executed?

Comment 7

[Greg Grisco]

(In reply to Nicolas B. Pierron [:nbp] from comment #6)
> As Bug 1017757, this stack is not actionable and we are not sure if it is
> correct.
> The only thing that tell us is that we are in compiled code.
> 
> Greg, would it be possible to have some human-readable interpretation of the
> RIL communication (from the logcat) which is going on, such as we can figure
> out what are the last pieces of JavaScript which were executed?

Hi Nicolas, there are actually two logs attached to the extra file.  The first is the regular android log, followed by the radio log.  Looking at the timestamp, the last RIL message came several seconds before the crash, so you probably want to focus on the section just above "--------- log /dev/log/radio"

Comment 8

[Nicolas B. Pierron [:nbp]]

(In reply to Greg Grisco from comment #7)
> you probably want to focus on the section just above "--------- log
> /dev/log/radio"

Thanks I did not noticed, I thought the end of the file was the end of the logcat.

I see a lot of errors coming from the camera drivers is that expected?

> 05-28 06:33:07.834 19563 19563 I Gecko   : 1401238987839	Marionette	… takesScreenshot …

Do we have this screenshot available?

> 05-28 06:33:08.834 19563 19563 I GeckoDump: Crash reporter : Not online, postponing.
> 05-28 06:33:08.884 19563 19563 W OomLogger: following kill message may be a duplicate

Do we know which application crashed, is that the one which got OOM-killed?

Comment 9

[Chris Peterson [:cpeterson]]

nbp is investigating this 1.4 blocker.

Comment 10

[Inder]

Please ni @ggrisco/@tkundu for quick turnaround.

Comment 11

[Greg Grisco]

(In reply to Nicolas B. Pierron [:nbp] from comment #8)
> (In reply to Greg Grisco from comment #7)
> > you probably want to focus on the section just above "--------- log
> > /dev/log/radio"
> 
> Thanks I did not noticed, I thought the end of the file was the end of the
> logcat.
> 
> I see a lot of errors coming from the camera drivers is that expected?
> 
> > 05-28 06:33:07.834 19563 19563 I Gecko   : 1401238987839	Marionette	… takesScreenshot …
> 
> Do we have this screenshot available?

Unfortunately, no screenshots available.

> > 05-28 06:33:08.834 19563 19563 I GeckoDump: Crash reporter : Not online, postponing.
> > 05-28 06:33:08.884 19563 19563 W OomLogger: following kill message may be a duplicate
> 
> Do we know which application crashed, is that the one which got OOM-killed?

I'm going to upload another pair of minidump/.extra file where you can see that several apps are getting OOM-KILLED.

Comment 12

[Greg Grisco]

Attachment: decoded minidump of crash (#2)

Comment 13

[Greg Grisco]

Attachment: EXTRA file attachment (plus some logs) (#2)

Comment 14

[Chris Peterson [:cpeterson]]

nbp: do Greg's new minidump or log file provide any new clues? The crash stack traces (or lack thereof) look the same.

I don't know if this is related, but Inder said that the San Diego test team (re)discovered bug 964537, a crash in the OomLogger itself. This bug was fixed in Gecko 31 (for B2G 2.0), but not uplifted to Gecko 30 for B2G 1.4.

Comment 15

[Nicolas B. Pierron [:nbp]]

(In reply to Chris Peterson (:cpeterson) from comment #14)
> nbp: do Greg's new minidump or log file provide any new clues? The crash
> stack traces (or lack thereof) look the same.
> 
> I don't know if this is related, but Inder said that the San Diego test team
> (re)discovered bug 964537, a crash in the OomLogger itself. This bug was
> fixed in Gecko 31 (for B2G 2.0), but not uplifted to Gecko 30 for B2G 1.4.

From the SEGV address it looks like this could be bug 964537, but why would the stack would be such a mess?

From the logcat, I see that one of the child process crashed (and later that Nuwa is OOM killed), and that the parent process produce a lot of errors due to IPC connections issues.

###!!! [Parent][MessageChannel] Error: Channel error: cannot send/recv
06-14 16:35:32.341 28913 28913 I GeckoDump: Crash reporter : Not online, postponing.
06-14 16:35:32.391 28913 28913 I Gecko   : 
###!!! [Parent][MessageChannel] Error: Channel error: cannot send/recv
06-14 16:35:32.451 28913 28913 E OomLogger: [Kill]: send sigkill to 25041 (Communications), adj 400, size 100

Greg, what is the KeyMaster?  Every 5 seconds or so, it is created and it exists after failing.

06-14 16:36:48.331 27594 27594 E keystore: keystore keymaster could not be initialized; exiting

Also, one thing which surprise me is the fast rate at which the PID are increasing.  I do not think the phone has enough process open to justify such a climb, but every report from the keystore message report a new PID every 5 seconds:

06-14 16:35:26.291 26254
06-14 16:35:32.051 26362
06-14 16:35:36.401 26473
06-14 16:35:41.821 26545
06-14 16:35:46.551 26594
06-14 16:35:51.781 26635
06-14 16:35:56.831 26735
06-14 16:36:02.761 26822
06-14 16:36:08.461 26894
06-14 16:36:13.111 26944
06-14 16:36:17.531 27022
06-14 16:36:23.151 27122
06-14 16:36:28.031 27170
06-14 16:36:32.671 27294
06-14 16:36:38.141 27425
06-14 16:36:43.121 27527
06-14 16:36:48.331 27594
06-14 16:36:53.441 27665
06-14 16:36:59.091 27746
06-14 16:37:03.701 27827

I also believed that we already cycled multiple times through the PID list, as the Main process pid is 28913.

I wonder if the system is crashing because of the zombification of the phone.

When you notice such crash, would it be possible to capture the result of "adb shell ps" ?

Comment 16

[Inder]

With the fix from bug 964537, this issue does not reproduce. Marking this as a duplicate.