666598 - Analyze Yahoo Mail Beta crash minidump

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[David Mandelin [:dmandelin]]

Attachment: JITCode around point of crash

Fresh bug for sanity preservation purposes. Marking s-s for now just in case. Sheila, please read the last bullet in this comment!

The crash is:

https://crash-stats.mozilla.com/report/index/e959acb3-4744-4adb-9332-5fa602110622

It does have jitcode at the point of the crash, which I extracted using MSVC and attached. There is nothing confidential in this jitcode.

On first look, the call to CompileFunction indicates this is a call IC stub generated by generateFullCallStub. This apparently is called only if the call site is a fun.apply() or an arity mismatch. The former seems like a good bet given this is big frameworky JS. 

To me, the jitcode looks corrupt almost from the start:

18148938  mov         dword ptr [ebx+50h],2  
1814893F  mov         dword ptr [ebx-19h],ebx  
18148942  dec         edx  

ebx-19h isn't a reasonable address to store to, is it? And I don't think ebx is a reasonable thing to store, either. Not sure about the first instruction, though, it looks like it might be OK.

Note also that the crash has several add-ons. There are unfamiliar DLLs in the list so I am pretty sure there are binary components here.

Can we get more eyes on this, stat? In particular:

- Is the jitcode in the attachment possibly from a JM bug (e.g., some other IC repatching way off the right position), or is it almost certainly the result of memory corruption?

- Do we have any other data that might make us suspicious of any of the add-ons here?

- Does it make sense to send Herbie a build with write protection on jitcode to help find who might be corrupting it? 

- Or should we just turn that on in Nightly and let the jitcode corruption crashes come in? If we do that, note that they won't have a single signature, so we need something in place to help us find them among the rest.

Comment 1

[Robert Kaiser]

(In reply to comment #0)
> If we do that, note that they won't have a single
> signature, so we need something in place to help us find them among the rest.

I don't think we currently have anything in place for this - but given that js::mjit::EnterMethodJIT is the highest signature (except some hangs) in 5.0 data now, we really want some instrumentation on those MethodJIT crashes.

How will the data look that we get there? I'm pulling in ted here in case we can do something on the side of stackwalking or so that could help us in instrumentation.

Comment 2

[Luke Wagner [:luke]]

(In reply to comment #0)
> On first look, the call to CompileFunction indicates this is a call IC stub
> generated by generateFullCallStub. This apparently is called only if the
> call site is a fun.apply() or an arity mismatch.

Its also (and perhaps primarily) called when there is more than one callee for a given callsite.

Comment 3

[(not currently active) Ted Mielczarek]

We could pretty easily do another run of bug 627111. I believe dvander had a check in there for "invalid JIT code".

Comment 4

[David Anderson [:dvander] - inactive, e-mail if emergency]

Categorizer seems to crap out on this dump.

dvander@dvander:$ ./categorize e959acb3-4744-4adb-9332-5fa602110622.dump 
2011-06-23 13:28:43: minidump.cc:3759: INFO: Minidump opened minidump /home/dvander/Downloads/e959acb3-4744-4adb-9332-5fa602110622.dump
2011-06-23 13:28:43: minidump.cc:3804: INFO: Minidump not byte-swapping minidump
2011-06-23 13:28:43: minidump.cc:2707: INFO: MinidumpMemoryList has no memory region at 0x1fae2a2b
NO_JIT_MEMORY

Comment 5

[David Mandelin [:dmandelin]]

Update: David Anderson can reproduce this bug locally. And it is not bug 662132.

Comment 6

[Daniel Veditz [:dveditz]]

(In reply to comment #0)
> Note also that the crash has several add-ons. There are unfamiliar DLLs in
> the list so I am pretty sure there are binary components here.

There are no correlations between this stack and any add-on, and only very weak correlations with dlls. Pretty sure those just happen to be there on this stack, but you won't see them in stacks in the other bug.

Comment 7

[Herbie D]

I have the minidump, but the upload will not take it. I can send it via e-mail directly, about an 80 kb zip file.

Comment 8

[David Mandelin [:dmandelin]]

(In reply to comment #7)
> I have the minidump, but the upload will not take it. I can send it via
> e-mail directly, about an 80 kb zip file.

No, you don't need to--we have already fixed the bug. You definitely helped us out by providing more data, which prompted me to bug David Anderson about it, upon which time he surprisingly was able to duplicate it locally and fix it.

Comment 9

[Boris Zbarsky [:bzbarsky]]

dmandelin, where's the patch for this living?  Or is it going to get attached here sometime?

Comment 10

[David Mandelin [:dmandelin]]

It's in bug 655660. We don't need this one any more.