822438 - crash in js::Interpret or js::mjit::JaegerShot on wiadomosci.onet.pl

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, defect)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-05490125-c9f4-494b-96bd-72d112121217 .
============================================================= 

Top frames:

0 		@0x13c879fa 	
1 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2454
2 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:363
3 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:396
4 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5850
5 	xul.dll 	nsJSContext::CallEventHandler 	dom/base/nsJSEnvironment.cpp:1918
6 	xul.dll 	nsGlobalWindow::RunTimeoutHandler 	dom/base/nsGlobalWindow.cpp:9612
7 	xul.dll 	nsGlobalWindow::RunTimeout 	dom/base/nsGlobalWindow.cpp:9863
8 	xul.dll 	nsGlobalWindow::TimerCallback 	dom/base/nsGlobalWindow.cpp:10130
9 	xul.dll 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:473
10 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624
11 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:82
12 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:201
13 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:175
...

Top URLs:

1950 	http://wiadomosci.onet.pl/tylko-w-onecie/internauci-krytykuja-obame-udawany-placz,1,5332480,wiadomosc.html
1573 	http://wiadomosci.onet.pl/swiat/szokujaca-pomylka-mediow-ws-mordercy-zamknijcie-si,1,5332057,wiadomosc.html
1057 	http://www.onet.pl/
918 	http://wiadomosci.onet.pl/swiat/usa-masakra-w-szkole-w-connecticut-26-zabitych,1,5332025,wiadomosc.html
773 	http://www.facebook.com/
712 	http://wiadomosci.onet.pl/tylko-w-onecie/13-grudnia-bedzie-goraco-wolnosc-w-polsce-jest-zag,1,5328229,wiadomosc.html
683 	http://wiadomosci.onet.pl/tylko-w-onecie/fala-oburzenia-po-przepowiedni-o-koncu-swiata,1,5331416,wiadomosc.html
617 	http://wiadomosci.onet.pl/swiat/matka-nauczyla-go-strzelac-ja-zabil-jako-pierwsza,1,5332578,wiadomosc.html
603 	http://wiadomosci.onet.pl/kraj/marsz-pis-w-31-rocznice-wprowadzenia-stanu-wojenne,1,5330789,wiadomosc.html
566 	http://wiadomosci.onet.pl/swiat/usa-strzelanina-w-szkole-w-stanie-connecticut,1,5331851,wiadomosc.html

There's a lot of other similar URLs in the list. The site seems to be a news portal.


js::Interpret is a signature that usually is around 200 crashes per million ADU, but in the last few days it jumped to about 4 times that volume, 829 crashes per million ADU on Firefox 17.0.* desktop in yesterday's data, see https://crash-analysis.mozilla.com/rkaiser/2012-12-16/2012-12-16.firefox.17.explosiveness.html - as can be seen there, the rise started on 2012-12-15.
The rise seem to be pretty specific to 17 - due to its usual level, the signature is seen across the board, though.

More crash reports can be found at https://crash-stats.mozilla.com/report/list?signature=js%3A%3AInterpret%28JSContext%2A%2C%20js%3A%3AStackFrame%2A%2C%20js%3A%3AInterpMode%29 - they are pretty spread across Windows versions and Firefox versions (even Thunderbird and SeaMonkey in the mix).

Correlations don't point to anything really interesting, only the URLs do.

Comment 1

[Robert Kaiser]

Due to this rise, this is now topcrash #2 on 17.0.1 desktop over the last 3 days.

On beta 18 and aurora 19, there is an according jump in the js::mjit::JaegerShot signature correlated with the same URLs, apparently - see https://crash-stats.mozilla.com/report/list?signature=js%3A%3Amjit%3A%3AJaegerShot%28JSContext%2A%2C%20bool%29 for more of those crash reports.

Comment 2

[u279076]

I'll see if I can reproduce this with any of the given URLs.

Comment 3

[u279076]

I've been fumbling around on wiadomosci.onet.pl with Firefox 17.0.1, Flash 11.5, and Windows XP. I've also tried throwing some Flash videos and games into the mix since a couple comments seemed to mention that. A lot of the content on the website is geo-blocked so I haven't been able to test everything. Unfortunately, I have yet to experience any crashes. 

Can anyone provide information which might enable me to do more targeted testing?

Comment 4

[Marcia Knous [:marcia]]

Here are some addon correlations, although there is no versions available in the manual report for the first signature in 17:

js::Interpret(JSContext*, js::StackFrame*, js::InterpMode)|EXCEPTION_ACCESS_VIOLATION_EXEC (1243 crashes)
     28% (350/1243) vs.   9% (5372/57419) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
      6% (74/1243) vs.   0% (243/57419) [e-mail]

For Firefox 18:

js::mjit::JaegerShot(JSContext*, bool)|EXCEPTION_ACCESS_VIOLATION_EXEC (97 crashes)
     44% (43/97) vs.   7% (1131/15930) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
      8% (8/97) vs.   0% (24/15930) [e-mail]
      7% (7/97) vs.   1% (155/15930) {1018e4d6-728f-4b20-ad56-37578a4de76b} (Flagfox, https://addons.mozilla.org/addon/5791)
      6% (6/97) vs.   0% (23/15930) [e-mail]
      6% (6/97) vs.   0% (35/15930) [e-mail]
      6% (6/97) vs.   1% (174/15930) {DDC359D1-844A-42a7-9AA1-88A850A938A8} (DownThemAll!, https://addons.mozilla.org/addon/201)

The second signature shows Version 2.2.1 of Adblock as being the one that is more prevalent:

js::mjit::JaegerShot(JSContext*, bool)|EXCEPTION_ACCESS_VIOLATION_EXEC (97 crashes)
     44% (43/97) vs.   7% (1131/15930) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
          0% (0/97) vs.   0% (7/15930) 2.0.3
          0% (0/97) vs.   0% (2/15930) 2.1.1
          1% (1/97) vs.   0% (21/15930) 2.1.2
          0% (0/97) vs.   0% (1/15930) 2.1.3a.3552
          2% (2/97) vs.   0% (2/15930) 2.2
         41% (40/97) vs.   7% (1097/15930) 2.2.1
          0% (0/97) vs.   0% (1/15930) 2.2.2a.3595

Comment 5

[u279076]

Thanks Marcia. I now have Adblock Plus, YoutubeToALL Player, and iPlexToALL Player addons installed in my profile. Still fumbling around on wiadomosci.onet.pl with those add-ons installed has not been enough to trigger any crashes for me. I didn't have success crashing using the add-on functionality on some of the website's Flash content either.

Comment 6

[Robert Kaiser]

I didn't mention the ABP correlation because it sounded pretty weak, and it still does.

There probably was a JavaScript change (as the crashes are with JS) on the 15th on that site (if we're unlucky, in an ad run on that site), but I'd not expect to see ABP in correlations that way in such a case) - I wonder if we have any way of contacting someone from that site and they could tell us if they rolled out some JS changes that day.

Comment 7

[Marcia Knous [:marcia]]

Doesn't hurt to try it with Adblock Plus installed - it is a simple operation and may or may not yield results. But some people crashing were definitely hitting it with that installed, and as you note it may be a website issue that triggered it. But we might need someone who speaks Polish to do that in order to be effective.

Comment 8

[Alex Keybl [:akeybl]]

(In reply to Robert Kaiser (:[e-mail]) from comment #6)
> I didn't mention the ABP correlation because it sounded pretty weak, and it
> still does.
> 
> There probably was a JavaScript change (as the crashes are with JS) on the
> 15th on that site (if we're unlucky, in an ad run on that site), but I'd not
> expect to see ABP in correlations that way in such a case) - I wonder if we
> have any way of contacting someone from that site and they could tell us if
> they rolled out some JS changes that day.

I've filed bug 822725 for this.

Comment 9

[Robert Kaiser]

So, it looks like the immediate spike of this is over and actually was only on the 15th and 16th, they may have seen the issue themselves and pulled back the change again.
Still, those signatures remain topcrashes even back at the levels they were at before, so it would still be interesting to know what change they made that triggered this, as it could lead to a testcase and a way we can fix a problem.

Comment 10

[Jacek Piskozub]

I posted an Firefox crashing URL and two possible JavaScript culprit files  received from onet.pl developers in this comment:
https://bugzilla.mozilla.org/show_bug.cgi?id=822725#c8

Comment 11

[Alex Keybl [:akeybl]]

Wontfixing for FF18 since the developers have pulled back the crashing pages. We should really try to resolve this in the FF19 timeframe, however, now that we have the suspected culprits.

Comment 12

[David Anderson [:dvander] - inactive, e-mail if emergency]

Looking at the signatures, the crash is manifesting somewhere in JIT code.

Jacek, would it be possible for the developers to restore the crashing page somewhere, or to give us a reproducible test case?

Comment 13

[Robert Kaiser]

(In reply to David Anderson [:dvander] from comment #12)
> Looking at the signatures, the crash is manifesting somewhere in JIT code.
> 
> Jacek, would it be possible for the developers to restore the crashing page
> somewhere, or to give us a reproducible test case?

See what Jacek pointed to in comment #10: There's some links in https://bugzilla.mozilla.org/show_bug.cgi?id=822725#c8 that should help you significantly there.

Comment 14

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Robert Kaiser (:[e-mail]) from comment #13)
> (In reply to David Anderson [:dvander] from comment #12)
> > Looking at the signatures, the crash is manifesting somewhere in JIT code.
> > 
> > Jacek, would it be possible for the developers to restore the crashing page
> > somewhere, or to give us a reproducible test case?
> 
> See what Jacek pointed to in comment #10: There's some links in
> https://bugzilla.mozilla.org/show_bug.cgi?id=822725#c8 that should help you
> significantly there.

That page doesn't crash anymore, it sounds like whatever exposed the bug got pulled from production.

Comment 15

[u279076]

Dropping qawanted from this bug since we've been unable to find steps to reproduce. The testcase in comment 10 does not reproduce anymore so I fear we might have dead-ended. Please advise if there is something more QA can do. Perhaps we could get onet.pl to stage the previously crashy code somewhere?

Comment 16

[Robert Kaiser]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #15)
> Perhaps we could get onet.pl to stage the previously crashy code
> somewhere?

Jacek, can you help there with your contact?

Comment 17

[Jacek Piskozub]

I'll try after the New Year. Now with school off, it could be a waste of time. Especially as I still do not have a direct access to them. he email I got was through their vanilla customer service. Therefore, I have the feeling that with key personnel on holidays my message could get stuck in a wrong department.

Comment 18

[Marcia Knous [:marcia]]

Jacek: Now that it is the New Year, can you follow up with someone in customer service? We are still seeing crashes in the Firefox 18 betas and we are getting close to the final release.  Thanks in advance for any help you can provide.

(In reply to Jacek Piskozub from comment #17)
> I'll try after the New Year. Now with school off, it could be a waste of
> time. Especially as I still do not have a direct access to them. he email I
> got was through their vanilla customer service. Therefore, I have the
> feeling that with key personnel on holidays my message could get stuck in a
> wrong department.

Comment 19

[Robert Kaiser]

Jacek, did you follow up with them?

Comment 20

[Alex Keybl [:akeybl]]

(In reply to Robert Kaiser (:[e-mail]) from comment #19)
> Jacek, did you follow up with them?

KaiRo - can you verify that the URLs remain the same?

Comment 21

[Robert Kaiser]

(In reply to Alex Keybl [:akeybl] from comment #20)
> KaiRo - can you verify that the URLs remain the same?

From all I know, the spike has gone away as they took back a change to their site, the URLs we had for the spike are the ones for the spike.
The question is just if they actually can give us a sample of the code they know triggered the crash or any information they have on how they triggered the crash, so we can debug our code that actually crashed.

Comment 22

[Alex Keybl [:akeybl]]

(In reply to Robert Kaiser (:[e-mail]) from comment #21)
> (In reply to Alex Keybl [:akeybl] from comment #20)
> > KaiRo - can you verify that the URLs remain the same?
> 
> From all I know, the spike has gone away as they took back a change to their
> site, the URLs we had for the spike are the ones for the spike.
> The question is just if they actually can give us a sample of the code they
> know triggered the crash or any information they have on how they triggered
> the crash, so we can debug our code that actually crashed.

Can we remove the topcrash keyword and untrack in that case? If yes, please do. I was confused by the fact that these signatures appeared to be associated with the #11 and #14 top crashers for FF18.

Comment 23

[Robert Kaiser]

(In reply to Alex Keybl [:akeybl] from comment #22)
> Can we remove the topcrash keyword and untrack in that case? If yes, please
> do. I was confused by the fact that these signatures appeared to be
> associated with the #11 and #14 top crashers for FF18.

The signature is pretty generic and loops in all kinds of crashes happening in JITed code, so yes, that's why the signature itself is in topcrash ranges. This particular variant isn't any more, thanks to Scoobidiver for removing the keyword.

On tracking, we can go either way. If onet lands this code again, we'll spike again, and if any other high-volume site does and the same code pattern (whatever it is), we'll also spike.
So, what is really important here is to get a code sample of the changes that lead to the crashes.

Comment 24

[Zibi Braniecki [:zbraniecki][:gandalf]]

Do we have a particular request for the onet dev team that would help us fix that?

The comments in bug 822725 indicate that we got in touch and we got the links. Is there something more that we want now?

Comment 25

[Robert Kaiser]

(In reply to Zbigniew Braniecki [:gandalf] from comment #24)
> The comments in bug 822725 indicate that we got in touch and we got the
> links. Is there something more that we want now?

The problem is that as they rolled back their change, the links point to versions of the code that doesn't trigger the crash, see comment #14. Jacek apparently only did get to "their vanilla customer service", see comment #17.

If we could reach their actual developers or otherwise find a way to get a hold of the version of the code that triggered the crashes, that would be awesome.

Comment 26

[Alex Keybl [:akeybl]]

No longer a topcrash, so no need to track for upcoming releases. We should continue outreach to onet.pl to find a reproducible testcase.

Comment 27

[Robert Kaiser]

gandalf, did you try to contact and get more information from them?

Comment 28

[Robert Kaiser]

Let's close this up as we removed JägerMonkey in 23 and therefore any crashes would happen with different signatures. Also, the new JIT might create different code and not crash at all.