1205771 - Firefox crashes on http://community.reactos.org/index.php/news

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect)

Description

[Acheron]

Attachment: 1442511703218.0986c522-3daf-4caf-8d75-50b23e5574e0.crash.jsonlz4

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0
Build ID: 20150826023504

Steps to reproduce:

Browse to http://community.reactos.org/index.php/news directly, by directly entering the url or by clicking on bookmark button.



Actual results:

Firefox crashes when loading the page every time. Note that when visiting http://communtity.reactos.org and clicking on the news link does not crash Firefox.


Expected results:

Firefox should display the web page.

Comment 1

[Acheron]

The correct link is actually http://community.reactos.org

Comment 2

[Loic]

WFm, surely graphics crash.

Type about:support in Firefox and copy the section "graphics".

Comment 3

[Acheron]

Here is the graphics section:

Adapter-RAM	Unknown
Adapterbeschrijving	Intel(R) G41 Express Chipset
Adapterstuurprogramma’s	igxprd32
Asynchroon pannen/zoomen	geen
Datum stuurprogramma	4-21-2010
Device-ID	0x2e32
Direct2D ingeschakeld	Geblokkeerd voor uw grafische stuurprogramma.
DirectWrite ingeschakeld	false (0.0.0.0)
GPU #2 actief	false
GPU-versnelde vensters	0/1 Basic Geblokkeerd voor uw grafische stuurprogramma.
Ondersteunt hardwarematige H264-decodering	false
Stuurprogrammaversie	6.14.10.5259
Subsys-ID	00000000
Vendor-ID	0x8086
WebGL-renderer	Geblokkeerd voor uw grafische stuurprogramma.
windowLayerManagerRemote	false
AzureCanvasBackend	skia
AzureContentBackend	cairo
AzureFallbackCanvasBackend	cairo
AzureSkiaAccelerated	0

Comment 4

[Acheron]

I have updated the graphics driver to 6.14.10.5420.

However it does not solve the crash. Firefox crashes on http://community.reactos.org/index.php/news when you force refresh the page.

Adapter-RAM	Unknown
Adapterbeschrijving	Intel(R) G41 Express Chipset
Adapterstuurprogramma’s	igxprd32
Asynchroon pannen/zoomen	geen
Datum stuurprogramma	7-23-2012
Device-ID	0x2e32
Direct2D ingeschakeld	Geblokkeerd voor uw grafische stuurprogramma.
DirectWrite ingeschakeld	false (0.0.0.0)
GPU #2 actief	false
GPU-versnelde vensters	0/1 Basic Geblokkeerd voor uw grafische stuurprogramma.
Ondersteunt hardwarematige H264-decodering	false
Stuurprogrammaversie	6.14.10.5420
Subsys-ID	00000000
Vendor-ID	0x8086
WebGL-renderer	Geblokkeerd voor uw grafische stuurprogramma.
windowLayerManagerRemote	false
AzureCanvasBackend	skia
AzureContentBackend	cairo
AzureFallbackCanvasBackend	cairo
AzureSkiaAccelerated	0

Comment 5

[Loic]

Did it use to crash with the previous versions of Firefox?

Comment 6

[Acheron]

I don't know as I have not visited the site often. Also I first noticed this issue when I bookmarked this url.

Comment 7

[Loic]

IMHO, the issue is about Intel(R) G41 Express Chipset, it's not the 1st bug with this GPU (bug 1186755).
Did you try with HWA disabled in the options?
http://support.mozilla.org/en-US/kb/forum-response-disable-hardware-acceleration

Comment 8

[Acheron]

The crash also happens using Firefox ESR 38.2 on a fresh profile using the same steps.

https://crash-stats.mozilla.com/report/index/7dae6889-7990-410d-84ef-e5fd32150917

Comment 9

[Loic]

On the FTP, you can find the old releases: http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/
Could you try with FF36 or 34, please.

Comment 10

[Acheron]

I can report success! The crash does not happen on Firefox 34. Both with HWA enabled and disabled (I have restarted the browser after changing the setting).

Comment 11

[Loic]

Well, there is maybe a regression since FF35 (even I don't know if it's possible to fix it).
Now we need to find a regression range with the nightly builds.

You can download the nightlies from FF34, they started in 2014-09:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/09/

For each day, there is a folder called "mozilla-central" and inside you can download the daily nightly (as *en-US.win32.zip, it's a standalone version, no install, just unzip and click on the .exe).

As FF34 nightlies started in Sept 2014, you can start by the nightlies of Nov 2014 (FF36), then you divide the range by 2 (dichotomy).

When you find the LAST good build (no crash/freeze according to your tests) and the FIRST bad build, you type about:buildconfig in the URL bar and copy here the link after "Built from" for both builds.

It's not very long to test, maybe 10-15 min. It would be helpful for the devs to narrow down the regressing patch.

Comment 12

[Acheron]

Firefox 36 crashes using the exact same steps both with HWA enabled and disabled.

https://crash-stats.mozilla.com/report/index/cb84b0ab-e654-4846-a078-c389b2150917

I don't think this is a display issue, as the site does display correctly on my monitor. The problem occurs when loading the site.

Comment 13

[Acheron]

Firefox 35 also crashes with HWA enabled and disabled. I will now test the nightlies between Fx 34 and 35.

Comment 14

[Acheron]

Here are the test results:

36.0.a1 2014-11-04 works (version 36.0.0.5421)
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-11-04-03-02-02-mozilla-central/ 
about:buildconfig

Built from https://hg.mozilla.org/mozilla-central/rev/5dde8ea48fef

36.0.a1 2014-11-05 firefox crashes (version 36.0.0.5422)
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-11-05-03-02-03-mozilla-central/

about:buildconfig

Built from https://hg.mozilla.org/mozilla-central/rev/62990ec7ad78

Comment 15

[Acheron]

During testing I also found that nightly builds 36.0.a1 2014-11-16 and newer do trap the catch and display the following message:

Tab crashed

Well, this is embarrassing. We tried to display this Web page, but it's not responding.

Comment 16

[Loic]

e10s was probably enabled. You can disable e10s in the options about:preferences where there is a checkbox to uncheck.

Comment 17

[Loic]

(e10s is the multiprocess feature, that's why only the tab crashed and not the entire browser)

Comment 18

[Loic]

According to your regression search, the pushlog is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5dde8ea48fef&tochange=62990ec7ad78

Comment 19

[Acheron]

I did not test newer Firefox releases, but the crash does not occur on Firefox 41 beta 9 and Firefox 43 nightlies. I will now try to find the exact nightly which fixes the issue.

Comment 20

[Acheron]

After I installed Firefox 41 beta 1 it crashed again on the url. Somehow Firefox 41 beta 9 and Firefox 43 nightly started having the same behavior. The browser also crashes when Firefox is started loading the last tab without forced refresh.

Comment 21

[Acheron]

More testing. On a clean profile Firefox is harder to crash. Firefox 40.0.3 needs 3 or 4 forced refreshes on the url to make it crash. After the first crash happened it crashes on the first forced refresh, or when the browser is trying to load the last url it crashes without a forced refresh.

Comment 22

[Loic]

 knekeman, could you confirm the changelog? (maybe do a confirmation run like "mozregression --bits=32 --good=2014-11-03 --bad=2014-11-06" to test only the last good and first bad)

Comment 23

[Acheron]

What information do you need exactly? 

The problem starts somewhere between nightly 2014-11-04 and 2014-11-05. 

I am willing to set up a debug environment, but I have no experience with building Firefox on Windows.

Comment 24

[Loic]

Yes, sorry for my last post, I thought you had installed the tool Mozregression (http://mozilla.github.io/mozregression/) to find a regression range, it's a python script to automatize the download and the launch of builds (with a bunch of features).

Comment 25

[Acheron]

Here are the test results using mozregression --bits=32 --good=2014-11-03 --bad=2014-11-06

e-13a3a9e97384}
 1:07.53 LOG: MainThread mozversion INFO application_name: Firefox
 1:07.53 LOG: MainThread mozversion INFO application_remotingname: firefox
 1:07.53 LOG: MainThread mozversion INFO application_repository: https://hg.mozi
lla.org/mozilla-central
 1:07.53 LOG: MainThread mozversion INFO application_vendor: Mozilla
 1:07.53 LOG: MainThread mozversion INFO application_version: 36.0a1
 1:07.53 LOG: MainThread mozversion INFO platform_buildid: 20141104030202
 1:07.53 LOG: MainThread mozversion INFO platform_changeset: 5dde8ea48fef
 1:07.53 LOG: MainThread mozversion INFO platform_repository: https://hg.mozilla
.org/mozilla-central
 1:07.55 LOG: MainThread mozversion INFO platform_version: 36.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', 'back' or 'exit' and press Enter): good
 2:09.59 LOG: MainThread Bisector INFO Narrowed nightly regression window from [
2014-11-03, 2014-11-05] (2 days) to [2014-11-04, 2014-11-05] (1 days) (~0 steps
left)
 2:09.59 LOG: MainThread main INFO Got as far as we can go bisecting nightlies..
.
 2:09.59 LOG: MainThread Bisector INFO Last good revision: 5dde8ea48fef
 2:09.59 LOG: MainThread Bisector INFO First bad revision: 62990ec7ad78
 2:09.59 LOG: MainThread Bisector INFO Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5dde8ea48fef&tocha
nge=62990ec7ad78

 2:09.61 LOG: MainThread Bisector INFO ... attempting to bisect inbound builds (
starting from 4 days prior, to make sure no inbound revision is missed)
 2:10.81 LOG: MainThread main INFO Getting inbound builds between 6bd2071b373f a
nd 62990ec7ad78

mozregression doesn't download any inbound builds. It just hangs at the last step.

Comment 26

[Loic]

Thanks for that, it confirms your manual bisection!
Yes, it hangs because it doesn't find any inbound builds, that's normal (especially with old nightlies) and after a while, it stops the search and exits.

Comment 27

[Acheron]

Is it necessary to do further research on this issue?

I have not been able to reproduce the crash issue on a fresh profile on Firefox 41 beta and latest Firefox 43 nightly. I only got crashes after loading a profile which had crashed before using an older Firefox release.

Comment 28

[Acheron]

Using the command mozregression --good-release 41 --bad-release 40 --find-fix

I have been able to get the following:

16:43.16 LOG: MainThread mozversion INFO platform_repository: https://hg.mozilla
.org/integration/mozilla-inbound
16:43.16 LOG: MainThread mozversion INFO platform_version: 41.0a1
Was this inbound build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', 'back' or 'exit' and press Enter): good
17:40.59 LOG: MainThread Bisector INFO Narrowed inbound regression window from [
b563fdf0, 8225a3b7] (3 revisions) to [b563fdf0, 223975f9] (2 revisions) (~1 step
s left)
17:40.59 LOG: MainThread main INFO Oh noes, no (more) inbound revisions :(
17:40.59 LOG: MainThread Bisector INFO First good revision: 223975f99ca3ec887c80
fc009426fffb8720af4d
17:40.59 LOG: MainThread Bisector INFO Last bad revision: b563fdf07f83c1978cacb6
e03df450575e9cbacd
17:40.59 LOG: MainThread Bisector INFO Pushlog:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b563fd
f07f83c1978cacb6e03df450575e9cbacd&tochange=223975f99ca3ec887c80fc009426fffb8720
af4d

Here is the pushlog:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b563fdf07f83c1978cacb6e03df450575e9cbacd&tochange=223975f99ca3ec887c80fc009426fffb8720af4d

Comment 29

[u279076]

(In reply to knekeman from comment #27)
> Is it necessary to do further research on this issue?
> 
> I have not been able to reproduce the crash issue on a fresh profile on
> Firefox 41 beta and latest Firefox 43 nightly. I only got crashes after
> loading a profile which had crashed before using an older Firefox release.

I suggest this bug can be closed as WONTFIX in this case since Firefox 41 is due to be released in a few days. Ritu, what do you think?

Comment 30

[Acheron]

I cannot reproduce the crash anymore in Firefox 41 beta and Firefox nightly release using a previously crashed profile.

Comment 31

[Acheron]

As noone has been able to reproduce the crash, I would suggest having this one listed as VERIFIED WORKSFORME and products affected Firefox 35 - 40.

Comment 32

[Acheron]

Looking for a correct way to handle a bug that has not been confirmed, but which is already fixed

I looked up the information in https://developer.mozilla.org/en/docs/What_to_do_and_what_not_to_do_in_Bugzilla

it says the following:

Resolve a bug as FIXED if the bug has been fixed by a checkin into the Mozilla Mercurial code repository. Bugs which can no longer be reproduced should be marked WORKSFORME instead of FIXED if they can't be linked to a single checkin.

I would like to have this issue marked as FIXED by linking it to a specific changeset.

Comment 33

[Acheron]

More work on why Firefox crashes on the specific url I saved the site to my computer for further research.

The same crash happens when loading the local webpage, so in case the site is modified in a way that the crash no longer occurs I still can provide you a testcase. As I do not own the code or any content on it I can not post it here.

I installed the Firefox Addon QuickJava to determine exactly which Firefox component is responsible for the crash.

With loading images disabled and javascript disabled the crash still occurs, but with stylesheets disabled (CSS) it does not and displays the webpage.
I then tried all combinations.

The crash occurs immediately when activating the stylesheets option when the site is already loaded.

In the pushlog there is one specific css change about removing box-sizing:padding-box.

Inspecting within the Developer Tools one reference to padding-box is found in rokbox.css

.rokbox-wrapper .rokbox-outer .rokbox-row .rokbox-inner .rokbox-container {
    background: black none repeat scroll 0 0 padding-box;
    border-radius: 6px;
    box-shadow: 0 3px 7px rgba(0, 0, 0, 0.3);
    display: inline-block;
    margin: 20px 40px;
    max-width: 1200px;
    min-height: 100px;
    min-width: 100px;
    outline: medium none;
    position: relative;
    text-align: left;
    vertical-align: middle;
}

However when loading the site with access to rokbox.css blocked the crash still happens.

In the local webpage padding-box is not found, as it only records effective stylesheet elements. 
Removing the rokbox.css file makes no difference.

Comment 34

[Acheron]

I narrowed down the issue to the combination of these three css files:

http://community.reactos.org/templates/rt_paradigm/css-compiled/master-d5360004f368e4cb854754be88ab8f47.css
http://community.reactos.org/templates/rt_paradigm/css-compiled/thirdparty-k2-d5360004f368e4cb854754be88ab8f47.css
http://community.reactos.org/templates/rt_paradigm/css-compiled/menu.css

Comment 35

[Milan Sreckovic [:milan] (needinfo for best results)]

This is amazing triage, thanks everybody.  I'd like the bug left open for a bit longer, I want to understand what it was that broke it (out of those patches) and what it was that fixed it, and why it is that it shows up as OOM, but it's good news that it seems to be working.

Comment 36

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #29)
> (In reply to knekeman from comment #27)
> > Is it necessary to do further research on this issue?
> > 
> > I have not been able to reproduce the crash issue on a fresh profile on
> > Firefox 41 beta and latest Firefox 43 nightly. I only got crashes after
> > loading a profile which had crashed before using an older Firefox release.
> 
> I suggest this bug can be closed as WONTFIX in this case since Firefox 41 is
> due to be released in a few days. Ritu, what do you think?

Sure. That makes sense.

Comment 37

[u279076]

WONTFIX for Firefox 40, WORKSFORME for Firefox 41 and later.

Comment 38

[Acheron]

Attachment: reactos_testcase.zip

Comment 39

[Acheron]

I have done intensive research on this crash and have been stripping the webpage of it's content, removing css and html elements, just as long as the crash kept occuring. It might have something to do with the very long class element list attached to the body element on the original, as removing the long list resolved the crash issue. On my testcase I added an extra > character after the body element to get the crash.

During my research I found this is a very interesting bug, as it is dependent on so many things.

The crash only occurs when the content-type is set to UTF-8, you will need all the html elements,

and even the last lines with the font-size changes must be in a specific range (you can change some of them up or down at most 50%)

More interesting is that the crash only happens when the font-size specified in .catItemDateCreated::before is set to 0.9em and contains content: "";

Anything else and the crash does not happen.

These all have been taken from the original website.

Here's the stripped version in text:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  <title>community.reactos.org/index.php/news testcase</title>
  <link rel="stylesheet" href="menu.css" type="text/css">
</head>
<body class="font-size-is-default">>
  <div id="rt-page-surround">
    <ul>
      <li class="parent">
        <a class="item">test</a>
      </li>
    </ul>
    <span class="catItemDateCreated">Monday, 03 November 2014 20:19</span>
    <div class="catItemIntroText">
      <h2>
        The Party!
      </h2>
    </div>
    <div class="catItemIntroText">
      As you may know, JRE 7 has been the <i>Most voted app</i>
    </div>
    <h4>
      More...
    </h4>
    <h3>
      <a>test</a>
    </h3>
    <h2>
      Search
    </h2>
    <a href="#" style="font-size:150%">moonshot</a>
    <a href="#" style="font-size:100%">network</a>
    <a href="#" style="font-size:200%">perk</a>
    <a href="#" style="font-size:100%">polling</a>
    <a href="#" style="font-size:175%">story</a>
  </div>
</body>
</html>

with the following css:

.font-size-is-default { font-size: 15px; }

div#rt-page-surround { font-family: sans-serif; }

li.parent > .item::after { content: "test"; opacity: 0.5; }

.catItemDateCreated::before { font-family: SomeNonExistantFont; font-size: 0.9em; content: ""; color:rgba(0, 0, 0, 0.5);
}

.catItemIntroText { font-size: 12px; }

Comment 40

[Acheron]

Attachment: reactos_testcase_v2

Comment 41

[Acheron]

More work done on simplifying the code.

The extra > after the body tag is no longer required, and I have changed most elements to normal div ones.

The new html code:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  <title>community.reactos.org/index.php/news testcase</title>
  <link rel="stylesheet" href="menu.css" type="text/css">
</head>
<body class="font-size-is-default">
  <div class="catItemDateCreated">Monday, 03 November 2014 20:19</div>
  <div id="page-surround">
    <h2>The Party!</h2>
    <li><i>Most voted app</i></li>
  </div>
  <h2>More...</h2>
  <h3>test</h3>
  <h4>Search</h4>
  <div style="font-size:150%">moonshot</div>
  <div style="font-size:100%">network</div>
  <div style="font-size:200%">perk</div>
  <div style="font-size:100%">polling</div>
  <div style="font-size:175%">story</div>
</body>
</html>

and the css:

.font-size-is-default { font-size: 15px; }
.catItemDateCreated::before { font-size: 0.9em; content: ""; color: rgba(0, 0, 0, 0.5); }
li::after { content: "test"; opacity: 0.5; }
#page-surround { font-family: sans-serif; }

Comment 42

[Loic]

As the CSS code is small, you can directly insert it with the <style> tag in your html to avoid zip archive to attach.

Comment 43

[Acheron]

Attachment: reactos_testcase

Comment 44

[Acheron]

Attachment: reactos_crashtest.html

Comment 45

[Acheron]

I have created another testcase which can crash Firefox 41 and 44 nightly when zooming maximum out and maximum in multiple 1 or multiple times.

I have tested this on a fresh profile with all settings set to default. Interestingly Firefox nightly only crashes when electrolysis is off.

The crash does not occur on Windows 7 SP1 64 bit, or when booting Windows XP in VGA-mode.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  <title>community.reactos.org/index.php/news testcase</title>
  <style>
  #catItemDateCreated::before { /* crash also happens when using ::after */
	font-size: 11px; /* crashes on font-size values 6 px to 11 px, non-integer values 0.6 em to 0.8 em, pt values between 5pt and 9.5pt, when zooming in and out (1 or more times) depending on amount of text in second div */
	content: ""; /* crashes on U+F016, U+F017, U+F018, U+F01C,
					U+F02D, U+F02E, U+F02F, U+F030,
					U+2588, U+253C, U+256C,
					U+2591, U+2592, U+2593 */
 	color: rgba(0, 0, 0, 0.5); /* Crashes on any value between 0 and 1.
	                              Crashes only happen when changing opacity through color option.  */
  }
  #most-voted::after { content: "test"; opacity: 0.5; } /* Note that switching opacity and color option around does not cause a crash. Crash also happens when using ::before, or when applied to catItemDateCreated without second div or additional text */
  </style>
</head>
<body>
<div id="catItemDateCreated">Monday, 03 November 2014 20:19</div>
<div id="most-voted">Bush hid the facts</div>
</body>
</html>

Comment 46

[Acheron]

Attachment: reactos_crashtest.html

Comment 47

[Loic]

If you want to search for a regression range, there is the tool Mozregression: http://mozilla.github.io/mozregression/

Don't forget to set --bits=32 if you are on Windows.

Comment 48

[Acheron]

The new testcase crashes also on Firefox 34. Then I tested Firefox 4.0 and 16.0 which did not crash on the testcase. So it is still a regression!

Here's the output of mozregression, which points to a Firefox 27 nightly build.

 5:14.01 LOG: MainThread mozversion INFO platform_version: 27.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', 'back' or 'exit' and press Enter): good
 5:34.69 LOG: MainThread Bisector INFO Narrowed nightly regression window from [
2013-09-19, 2013-09-21] (2 days) to [2013-09-20, 2013-09-21] (1 days) (~0 steps
left)
 5:34.69 LOG: MainThread main INFO Got as far as we can go bisecting nightlies..
.
 5:34.69 LOG: MainThread Bisector INFO Last good revision: 8f8a683dfc42
 5:34.69 LOG: MainThread Bisector INFO First bad revision: a2c31dc69ab3
 5:34.69 LOG: MainThread Bisector INFO Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f8a683dfc42&tochange=a2c31dc69ab3

 5:34.69 LOG: MainThread Bisector INFO ... attempting to bisect inbound builds (
starting from 4 days prior, to make sure no inbound revision is missed)

Comment 49

[Acheron]

I noticed Firefox does not crash on my latest testcase when using e10s mode. E10S mode is available since 2011, but by default it is not enabled in the nightlies until 2014-11-16.

So I created a test-profile with browser.tabs.remote and browser.tabs.remote.autostart set to true and run mozregression again with the following command:

mozregression --good=2014-11-16 --bad=2013-09-21 --find-fix

 6:02.08 LOG: MainThread mozversion INFO platform_version: 30.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', 'back' or 'exit' and press Enter):
 6:14.16 LOG: ProcessReader Test Runner WARNING Process exited with code 3221225
477
bad
 6:17.89 LOG: MainThread Bisector INFO Narrowed nightly regression window from [
2014-02-13, 2014-02-11] (2 days) to [2014-02-12, 2014-02-13] (1 days) (~0 steps
left)
 6:17.91 LOG: MainThread main INFO Got as far as we can go bisecting nightlies..
.
 6:17.91 LOG: MainThread Bisector INFO First good revision: a62bde1d6efe
 6:17.91 LOG: MainThread Bisector INFO Last bad revision: 802d87c77e76
 6:17.91 LOG: MainThread Bisector INFO Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=802d87c77e76&tochange=a62bde1d6efe

Comment 50

[Milan Sreckovic [:milan] (needinfo for best results)]

https://crash-stats.mozilla.com/report/index/2d02f30a-db4f-415a-a5ff-55c422150903 (11 seconds in), https://crash-stats.mozilla.com/report/index/bed6d550-d672-435e-b04a-978422150911 (3 seconds in) are start up, or close enough to it.  David, anything we can do to guard against these?  Or are they perhaps fixed in later releases?

Bas, I still don't believe in a simple OOM answer.  The fact that it's the basic layers combined with accelerated OMTC makes me think there is some race condition we're not taking care of.  Bug 805406, and since that one we had a few more.  Let's get to the bottom of this, and at least put in instrumentation so that we can be sure it is OOM, and tag exactly where that happens.  Or put more critical errors to annotate and get more info.

Comment 51

[Acheron]

I have run more regression tests to determine the exact nightly range the crash occurs with E10S enabled and disabled on my latest testcase and on http://community.reactos.org/index.php/news

First the results for my testcase (stripped down website, attached to this bug report):

With E10S disabled:

Crashes on the testcase when zooming in and out 1 or more times.

The crash occurs starting from 2013-09-21 (2013-09-20 was good) up until today (2015-09-29):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a2c31dc69ab3&tochange=8b4d14afc4f6

With E10S enabled:

The crash also occurs on the testcase, but is much harder to produce. When the crash does not occur directly,
it often helps to browse to some other websites and then reload the page again, 
zooming out and in multiple times.

The crash starts happening at the same nightly 2013-09-21, but was first fixed three weeks later in nightly 2013-10-08 (2013-10-07 is bad):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8b4d14afc4f6&tochange=f97307cb4c95

During testing I found the crash with E10S enabled was back almost a year later in nightly 2014-09-19 (2014-09-18 was good) and occurs when zooming in and out, now the behavior is the same as when E10S disabled:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=426497473505&tochange=c8e325eee9e1

However, the crash does not occur anymore in E10S mode starting from nightly 2014-11-16 (2014-11-15 is bad):
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=acbd7b68fa8c&tochange=a52bf59965a0

Then the original problem, loading the website http://community.reactos.org/index.php/news

Firefox crashes immediately when loading the website without zooming in or out.

Interestingly in this case the nightly affected builds range is not different for E10S enabled or disabled.

The crash appears at nightly 2014-11-05 (2014-11-04 was good), more than a year later than my latest testcase:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5dde8ea48fef&tochange=62990ec7ad78

The crash is fixed between nightly 2015-05-28 and 2015-05-29. Inbound builds are still available, and I could 
narrow the revision range to 5 changes:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b563fdf07f83c1978cacb6e03df450575e9cbacd&tochange=223975f99ca3ec887c80fc009426fffb8720af4d

Comment 52

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Milan Sreckovic [:milan] from comment #50)
> https://crash-stats.mozilla.com/report/index/2d02f30a-db4f-415a-a5ff-
> 55c422150903 (11 seconds in),
> https://crash-stats.mozilla.com/report/index/bed6d550-d672-435e-b04a-
> 978422150911 (3 seconds in) are start up, or close enough to it.  David,
> anything we can do to guard against these?  Or are they perhaps fixed in
> later releases?

Since this is just a general draw call that can happen anywhere, it is not so easy to use the crash guard. And the crash guard's "fix" is to disable acceleration - which is already disabled. There's nothing to fall back to here AFAICT.

Comment 53

[Acheron]

Here is the correct link to the pushlog for nightly 2013-10-07/2013-10-08:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5f0569c3cb8f&tochange=56b0a41985f3

containing the following interesting fix:

Bug 922593 - Prevent signed integer overflows calculating gaussian blurs

Comment 54

[Bas Schouten (:bas.schouten)]

(In reply to Milan Sreckovic [:milan] from comment #50)
> https://crash-stats.mozilla.com/report/index/2d02f30a-db4f-415a-a5ff-
> 55c422150903 (11 seconds in),
> https://crash-stats.mozilla.com/report/index/bed6d550-d672-435e-b04a-
> 978422150911 (3 seconds in) are start up, or close enough to it.  David,
> anything we can do to guard against these?  Or are they perhaps fixed in
> later releases?
> 
> Bas, I still don't believe in a simple OOM answer.  The fact that it's the
> basic layers combined with accelerated OMTC makes me think there is some
> race condition we're not taking care of.  Bug 805406, and since that one we
> had a few more.  Let's get to the bottom of this, and at least put in
> instrumentation so that we can be sure it is OOM, and tag exactly where that
> happens.  Or put more critical errors to annotate and get more info.

These are two completely different crashes that seem the have nothing in common except the function on the top of the stack.

Note the first is: EXCEPTION_ILLEGAL_INSTRUCTION, meaning the stack has to be corrupted there in any case. We must've somehow made an illegal jump or hit an alignment issue. Quite mysterious, and not really anything diagnostic that can be done.

The second is an EXCEPTION_ACCESS_VIOLATION_WRITE, with a corrupted bottom of the stack as well. But it's also in a totally different line of the function.

Neither of those have anything to do with the (indeed strange) OOM in this bug.

About the 'OOM' in this bug, the only explanation I have looking at the code is that cairo somehow in an earlier call got asked to allocate something really big which but the surface in an error status that was never cleared. This would call all subsequent CreateSimilar calls to fail (this is simply part of how cairo works).

I'm not certain if there's any way to clear the error status of a cairo surface, not that I know of, do you know Jeff?

Comment 55

[Jeff Muizelaar [:jrmuizel]]

(In reply to Bas Schouten (:bas.schouten) from comment #54)
> 
> I'm not certain if there's any way to clear the error status of a cairo
> surface, not that I know of, do you know Jeff?

No I don't think that's possible.

Comment 56

[Bas Schouten (:bas.schouten)]

A stack trace from this build would be helpful:

https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-97c63b39e0f8

Comment 57

[Bas Schouten (:bas.schouten)]

Build is done.

Comment 58

[Acheron]

Attachment: firefox-debug_0b6c_2015-09-30_23-35-26-625.log

windbg stacktrace nightly 2015-09-29

Comment 59

[Acheron]

The Firefox try debug build does not run on my system, and I could not get a stack trace using the non-debug build. There was also a problem loading the symbols.

But I managed to get a stack trace using nightly 2015-09-29, crashing in non-ES10 mode.

Comment 60

[Acheron]

Attachment: reactos_crashtest.html

Comment 61

[Bas Schouten (:bas.schouten)]

(In reply to knekeman from comment #59)
> The Firefox try debug build does not run on my system, and I could not get a
> stack trace using the non-debug build. There was also a problem loading the
> symbols.
> 
> But I managed to get a stack trace using nightly 2015-09-29, crashing in
> non-ES10 mode.

That's not super-useful, we really need one from my real debug build (it contained additional debugging), try installing the MSVC Debug redistributables and running it then, it's probably just missing those DLLs

Comment 62

[Acheron]

I have done some more tests and have found that this problem is caused by the Times New Roman font.

The Times New Roman font 3.00 included with Windows XP crashes on the following characters:
										U+F016, U+F017, U+F018, U+F01C,
U+F02D, U+F02E, U+F02F, U+F030,
U+2588, U+253C, U+256C,
U+2591, U+2592 and U+2593

So I updated the Times New Roman font to 5.07 (from Windows 7), which reduces the list of problematic characters to the following 6:

U+2588, U+253C, U+256C,
U+2591, U+2592 and U+2593

I have updated the testcase to use character U+2588 instead.

During testing I also found that setting opacity is not needed to cause the crash on latest nightly. See the comments in the html file for more information.

Comment 63

[Acheron]

Attachment: firefox-debug_0b6c_2015-09-30_23-35-26-625_new.log

Comment 64

[Acheron]

Without the opacity set I got a more useful stack trace on my latest testcase using nightly 2015-09-30.

Comment 65

[Bas Schouten (:bas.schouten)]

(In reply to knekeman from comment #64)
> Without the opacity set I got a more useful stack trace on my latest
> testcase using nightly 2015-09-30.

Still not very useful, we really need to know where the actual cairo error occurs, the only way to get that information is through a specially instrumented build.

Comment 66

[Milan Sreckovic [:milan] (needinfo for best results)]

Given a crash of the optimized version of the "special build", we should be able to extract the stack based on Bas' source code - was there a crash submitted?

Comment 67

[Milan Sreckovic [:milan] (needinfo for best results)]

The VC redistributables that Bas is talking about in comment 61.

Install VC 2013 (http://www.microsoft.com/en-us/download/details.aspx?id=40784) & VC 2015 (https://www.microsoft.com/en-us/download/details.aspx?id=48145) runtimes for x86 and x64 and reboot.

Comment 68

[Acheron]

I have installed the runtime files and rebooted the computer. I also profiled the debug build using Dependency Walker but it doesn't show any VC files to be missing. 

While running the debug build I first get a command prompt window with the following:

#10: NS_UTF16ToCString[xul +0x1242ff]
#11: JSPrincipals::dump[xul +0xacd617]
#12: JSPrincipals::dump[xul +0xacceca]
#13: sh::ShaderVariable::isArray[xul +0x1c8ee6d]
#14: sh::ShaderVariable::isArray[xul +0x1c8f6d9]
#15: sh::ShaderVariable::isArray[xul +0x1d1a32c]
#16: sh::ShaderVariable::isArray[xul +0x1d1a7d7]
#17: sh::ShaderVariable::isArray[xul +0x1d19ffd]
#18: sh::ShaderVariable::isArray[xul +0x1ff980a]
#19: sh::ShaderVariable::isArray[xul +0x1fe0c3f]
#20: sh::ShaderVariable::isArray[xul +0x1fdcc36]
#21: sh::ShaderVariable::isArray[xul +0x1fdcd3a]
#22: sh::ShaderVariable::isArray[xul +0x203e510]
#23: sh::ShaderVariable::isArray[xul +0x203eae2]
#24: sh::ShaderVariable::isArray[xul +0x2039bc3]
#25: JS::ProfilingFrameIterator::jitIter[xul +0x220e98a]
#26: sh::ShaderVariable::isArray[xul +0x20097b2]
#27: sh::ShaderVariable::isArray[xul +0x2008b5f]
#28: XRE_GetFileFromPath[xul +0x2253042]
#29: XRE_GetFileFromPath[xul +0x2252592]
#30: XRE_GetFileFromPath[xul +0x22559bd]
#31: XRE_GetFileFromPath[xul +0x22539b8]
#32: XRE_main[xul +0x225645a]
#33: ???[firefox +0x259d]
#34: ???[firefox +0x1ed3]
#35: ???[firefox +0x2915]
#36: ???[firefox +0x4c89]
#37: RegisterWaitForInputIdle[kernel32 +0x16037]

followed by an access violation occurred in xul.dll at address 0x0223F520

Comment 69

[Acheron]

Here's a crash report generated by the non-debug special try build:

https://crash-stats.mozilla.com/report/index/82d15c4a-0b68-476b-a2cd-833da2151001

Comment 70

[Acheron]

I have tested the latest testcase also on Windows 7 64 bit (same machine), but the problematic characters U+2588, U+253C, U+256C, U+2591, U+2592 and U+2593 do not cause a crash on that OS. The text remains visible on all zoom levels.

Comment 71

[Bas Schouten (:bas.schouten)]

(In reply to knekeman from comment #68)
> I have installed the runtime files and rebooted the computer. I also
> profiled the debug build using Dependency Walker but it doesn't show any VC
> files to be missing. 
> 
> While running the debug build I first get a command prompt window with the
> following:
> 
> #10: NS_UTF16ToCString[xul +0x1242ff]
> #11: JSPrincipals::dump[xul +0xacd617]
> #12: JSPrincipals::dump[xul +0xacceca]
> #13: sh::ShaderVariable::isArray[xul +0x1c8ee6d]
> #14: sh::ShaderVariable::isArray[xul +0x1c8f6d9]
> #15: sh::ShaderVariable::isArray[xul +0x1d1a32c]
> #16: sh::ShaderVariable::isArray[xul +0x1d1a7d7]
> #17: sh::ShaderVariable::isArray[xul +0x1d19ffd]
> #18: sh::ShaderVariable::isArray[xul +0x1ff980a]
> #19: sh::ShaderVariable::isArray[xul +0x1fe0c3f]
> #20: sh::ShaderVariable::isArray[xul +0x1fdcc36]
> #21: sh::ShaderVariable::isArray[xul +0x1fdcd3a]
> #22: sh::ShaderVariable::isArray[xul +0x203e510]
> #23: sh::ShaderVariable::isArray[xul +0x203eae2]
> #24: sh::ShaderVariable::isArray[xul +0x2039bc3]
> #25: JS::ProfilingFrameIterator::jitIter[xul +0x220e98a]
> #26: sh::ShaderVariable::isArray[xul +0x20097b2]
> #27: sh::ShaderVariable::isArray[xul +0x2008b5f]
> #28: XRE_GetFileFromPath[xul +0x2253042]
> #29: XRE_GetFileFromPath[xul +0x2252592]
> #30: XRE_GetFileFromPath[xul +0x22559bd]
> #31: XRE_GetFileFromPath[xul +0x22539b8]
> #32: XRE_main[xul +0x225645a]
> #33: ???[firefox +0x259d]
> #34: ???[firefox +0x1ed3]
> #35: ???[firefox +0x2915]
> #36: ???[firefox +0x4c89]
> #37: RegisterWaitForInputIdle[kernel32 +0x16037]
> 
> followed by an access violation occurred in xul.dll at address 0x0223F520

That's really mysterious...

Comment 72

[Acheron]

Attachment: firefox-debug_0434_2015-10-02_13-29-09-875.log

Comment 73

[Acheron]

I have finally been able to get a stack trace using your special build. To get a stack trace I copied the pdb files from latest nightly.

The stack trace is different from latest nightly and points to xul!nsTreeBodyFrame::PaintCell

Comment 74

[Bas Schouten (:bas.schouten)]

You'll need to get the PDB files from the build I made.

http://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-97c63b39e0f8/

It should contain crash reporter symbols at least, not sure if you can get the original PDBs from that.

Comment 75

[Acheron]

(In reply to Bas Schouten (:bas.schouten) from comment #74)
> You'll need to get the PDB files from the build I made.
> 
> http://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/
> [e-mail]-97c63b39e0f8/
> 
> It should contain crash reporter symbols at least, not sure if you can get
> the original PDBs from that.

I have used your symbols, but they only contain .sym files within the .pdb folders. I actually need .pdb files within the .pdb folder.

Comment 76

[Acheron]

Attachment: reactos_crashtest_v4.html

Comment 77

[Acheron]

using the text-transform scale option I have been able to further reduce the testcase to the following:

<html>
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8"><!-- must be set to utf-8 or utf-16 -->
  <title>community.reactos.org/index.php/news testcase</title>
  <style>
	#div1 {
		font-family: "Georgia","Constantia","Times New Roman"; /* Crash happens on any font */
		transform: scale(1.05, 1.05);	/* Any font scaling causes a crash immediately or when zooming in or out */
	}
	#div2 {
		opacity: 0.5;	/* Crash happens on any value between 0 and 1.
						   Interestingly the crash does not happen when specifying RGBA or HSLA alpha channel colors instead.
						*/
	}
  </style>
</head>
<!--	/* When using Times New Roman 3.00 (Windows XP) the problematic characters are:
			U+F016, U+F017, U+F018, U+F01C,
			U+F02D, U+F02E, U+F02F, U+F030,
			U+2588, U+253C, U+256C,
			U+2591, U+2592 and U+2593
		   When using Times New Roman 5.07 (Windows 7) the problematic characters are:
			U+2588, U+253C, U+256C,
			U+2591, U+2592 and U+2593 */ -->
<!--	/* Crash occurs only when the problematic character is the first character of the element's content. Divs can be put in different order. */ -->
<body>
  <div id="div1">█ anything here</div>
  <div id="div2">Monday, 03 November 2014 20:19</div>
</body>
</html>

Interestingly this testcase is not dependent on the Times New Roman font anymore and more important is that it also crashes in E10S mode.

I also tested the new testcase on Windows 7 but the crash does not occur on that platform.

Comment 78

[Acheron]

Attachment: firefox-debug_0f40_2015-10-05_17-48-44-078.log

Comment 79

[Acheron]

I have managed to get a better stack trace by disabling all Firefox plugins first.

The interesting part:

ChildEBP RetAddr  
0012a994 115210b7 xul!nsTreeBodyFrame::PaintCell(int aRowIndex = 0n128175616, class nsTreeColumn * aColumn = 0x00000001, struct nsRect * aCellRect = 0x0012b5c4, class nsPresContext * aPresContext = 0x081a4e00, class nsRenderingContext * aRenderingContext = 0x0012adcc, struct nsRect * aDirtyRect = 0x06954d00, int * aCurrX = 0x06954d00, struct nsPoint aPt = struct nsPoint)+0x74d [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\layout\xul\tree\nstreebodyframe.cpp @ 3297]
0012aa48 11508a5f xul!nsTreeBodyFrame::PaintCheckbox(int aRowIndex = 0n128175616, class nsTreeColumn * aColumn = 0x00000002, struct nsRect * aCheckboxRect = 0x0012aac4, class nsPresContext * aPresContext = 0x00000000, class nsRenderingContext * aRenderingContext = 0x00000000, struct nsRect * aDirtyRect = 0x0012adcc)+0x128 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\layout\xul\tree\nstreebodyframe.cpp @ 3760]
0012bdc8 1152674f xul!mozilla::dom::BoxObject::GetScreenX(int * _retval = 0x00000000)+0x12 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\layout\xul\boxobject.cpp @ 277]
0012bdcc 081a4e00 xul!nsGrid::GetExtraColumnCount(bool aIsHorizontal = false)+0x3 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\layout\xul\grid\nsgrid.cpp @ 502]
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012bdf4 10516506 0x81a4e00
0012be00 00000000 xul!xpc::StackScopedCloneOptions::Parse(void)+0x25 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\js\xpconnect\src\xpcprivate.h @ 3569]

Comment 80

[Acheron]

Yet another pushlog for my latest testcase with E10S enabled.

 8:00.78 LOG: MainThread mozversion INFO platform_version: 35.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', 'back' or 'exit' and press Enter): good
 8:28.50 LOG: MainThread Bisector INFO Narrowed nightly regression window from [
2014-09-01, 2014-09-04] (3 days) to [2014-09-03, 2014-09-04] (1 days) (~0 steps
left)
 8:28.50 LOG: MainThread main INFO Got as far as we can go bisecting nightlies..
.
 8:28.50 LOG: MainThread Bisector INFO Last good revision: 5e9826980be5
 8:28.50 LOG: MainThread Bisector INFO First bad revision: 776fa9cf70cd
 8:28.50 LOG: MainThread Bisector INFO Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5e9826980be5&tochange=776fa9cf70cd

Comment 81

[Acheron]

The same crash also happens on a virtual machine running Windows XP. I have been able to get the Firefox Debug builds to run on this virtual machine, but I would like to generate a stack trace with symbols.

The debug symbols contain only .sym files within the pdb folders and I have not been able to get these to load in WinDBG. 

Bas, can you help me configure the symbols for these Debug builds?

Comment 82

[Bas Schouten (:bas.schouten)]

(In reply to knekeman from comment #81)
> The same crash also happens on a virtual machine running Windows XP. I have
> been able to get the Firefox Debug builds to run on this virtual machine,
> but I would like to generate a stack trace with symbols.
> 
> The debug symbols contain only .sym files within the pdb folders and I have
> not been able to get these to load in WinDBG. 
> 
> Bas, can you help me configure the symbols for these Debug builds?

I'm not sure how you can get PDB files from .sym files, sorry :(. You should probably ask someone who knows the crash reporter and such stuff a little better.

Comment 83

[(No longer employed by Mozilla) Aaron Klotz]

See the following url:
https://developer.mozilla.org/en/docs/Using_the_Mozilla_symbol_server

Then you can configure your debugger to automagically download the pdbs.

Comment 84

[Acheron]

Aaron, I already configured everything and I can actually use the symbols for Firefox release and nightly builds. However WinDBG doesn't find any symbols for either the Firefox debug builds or the custom builds by Bas Schouten. He explicitly asks for a stack trace for his build, as the regular builds don't provide enough information.

So is it normal that I don't get the symbols for the firefox debug builds (the ones from mozilla-central-debug)?

During my search for a solution I may have found the problem.

I found the following at https://wiki.mozilla.org/ReleaseEngineering/TryServer

By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 to the in-tree mozconfigs. You can do this for all platforms by importing this patch into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

Comment 85

[Bas Schouten (:bas.schouten)]

(In reply to knekeman from comment #84)
> Aaron, I already configured everything and I can actually use the symbols
> for Firefox release and nightly builds. However WinDBG doesn't find any
> symbols for either the Firefox debug builds or the custom builds by Bas
> Schouten. He explicitly asks for a stack trace for his build, as the regular
> builds don't provide enough information.
> 
> So is it normal that I don't get the symbols for the firefox debug builds
> (the ones from mozilla-central-debug)?
> 
> During my search for a solution I may have found the problem.
> 
> I found the following at
> https://wiki.mozilla.org/ReleaseEngineering/TryServer
> 
> By default native debug symbols are not uploaded for Try server builds
> because of their size. If you want to debug your builds locally you must add
> MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 to the in-tree mozconfigs. You can
> do this for all platforms by importing this patch into your mq and pushing
> it along with your changes to Try. This will cause a
> ...crashreporter-symbols-full.zip package to be uploaded to the builds
> directory for each platform built.

Hrm, shouldn't letting my debug build crash on your system just make the stack trace available through about:crashes?

Comment 86

[Acheron]

I have not been able to reproduce the crash since some time. The problem might be caused by a specific Windows Update (probably GDI or Win32k) and fixed later by installing newer GDI and Win32k security updates. I was able to get the same crash when running Windows XP in a VM, but that VM had also the current updates installed at that time.

Due to other stability issues I experienced on my system I have installed Windows XP again on a clean partition. Might the crash issue come back I will make sure that it is not introduced by Windows itself.

I'm currently running Firefox 43.0.3 without any stability issues.