911502 - Webfonts can cause system crash in Firefox 24.0b6+

Status: RESOLVED WORKSFORME

(Core :: Layout: Text and Fonts, defect)

Description

[Jeroen van der Gun]

Attachment: Info about my (test) Firefox installation

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0 (Beta/Release)
Build ID: 20130814063812

Steps to reproduce:

Do one of the following things after starting Firefox:
* Visit http://www.geeksphone.com/
* Visit https://support.mozilla.org/nl/questions/963226
* Open Firefox Health report
* ...


Actual results:

The computer crashes and reboots. (Hence Firefox crash reporter does not notice the crash.)

No crash occurs if gfx.downloadable_fonts.enabled is set to false. Whether JavaScript is enabled or not does not affect this bug; the same holds for hardware acceleration. Furthermore, Firefox 24.0b5 and earlier stable versions are not affected. I therefore conclude the crash is triggered by some change in the webfont system in Firefox 24.0b6 (and later).

However, it's not completely broken. I tried but failed to reduce the 'attack' to a simple proof-of-concept 'exploit'. Whenever I tried to do so the webfonts loaded correctly without crashes. Somehow webfonts work correctly in some cases, but crash the computer in others, and I haven't figured out why. Also, it appears that sometimes no crash occurs if the 'attack page' is not opened directly after a fresh browser start, but other pages are visited first - the reason is again unknown to me.

Comment 1

[Jeroen van der Gun]

Attachment: Video recording of a crash

Comment 2

[Mats Palmgren (inactive)]

These are the changes that landed between b5 and b6:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_24_0b5_RELEASE&tochange=FIREFOX_24_0b6_RELEASE
Nothing stands out as being related to downloadable fonts.

It would help a lot if you could test the builds in the range from here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/
(the folders with "beta" in the name, between Aug 22 to 26 or so)

Comment 3

[Mats Palmgren (inactive)]

It might be related to the crashes we have seen in the Nvidia driver
for some cards?  (since it crashes Windows)

Comment 4

[Jeroen van der Gun]

Below are my testing results for various builds. Somehow I can only reproduce the problem in beta release builds, not in debug builds nor in aurora/nightly builds.

Version                        Build ID        Source Stamp  Test Result
24.0b5                         20130822154523  abf43438122e  ok
24.0b6                         20130826142034  522e9b4cf952  affected
24.0b7                         20130829135643  0422783b9c2f  affected
24.0b 2013-08-24 debug         20130823155159  d856284278cc  ok
24.0b 2013-08-26 debug         20130825185524  b85f944181d8  ok
24.0b 2013-08-28 debug         20130827163217  8bded4829a2d  ok
24.0b 2013-09-01 debug         20130831090058  3181e9176487  ok
25.0a2 latest-mozilla-aurora   20130831004004  9dd5f0e7f716  ok
26.0a1 latest-mozilla-central  20130901030218  b6c29e434519  ok

Regarding Nvidia, I already updated my Nvidia driver to the version in attachment 798218 [details] before filing this bug, but that didn't affect the bug. Unfortunately, no blue screen of death appears with information. Note that I already disabled hardware acceleration in Firefox, but that doesn't prevent the crashes.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Can you try biecting within the list in:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-beta-win32/

(If the bug doesn't happen in the most recent of those, then there's no point.  But they ought to match what was in 24.0b5 and 24.0b6 more closely than the -debug nightlies, and the dates ought to line up as well, so the regression would most likely be in the August 22-26 range.)

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(Though, a note about the timestamps on those directories:  the timestamp in that directory list is the last time an automated test was run on that build, which could be substantially after it was built, but usually isn't.  The *name* of the directory is the timestamp of when it was built, in seconds since Jan 1, 1970, 00:00 UTC.  So they're probably (though not definitely) actually correctly sorted, even though it doesn't look like they are.  The SourceStamp inside the application.ini or in the .txt file next to the build is authoritative, though.)

Comment 7

[Jeroen van der Gun]

That is more effective. Below are the results for comment 5, with the official bètas also inserted in the timeline:

Timestamp   Build ID        Source Stamp  Test Result
1377198039  20130822120039  f935e4a1c6dd  ok
1377203736  20130822133536  3f1e67b70d5f  ok
[beta 5]    20130822154523  abf43438122e  ok
1377236663  20130822224423  1c39ca9daa6c  affected
1377298319  20130823155159  d856284278cc  affected
[beta 6]    20130826142034  522e9b4cf952  affected
1377559364  20130826162244  34fd3a39ab84  affected
[beta 7]    20130829135643  0422783b9c2f  affected

Looks like it went wrong on August 22.

Comment 8

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Jeroen van der Gun from comment #7)
> That is more effective. Below are the results for comment 5, with the
> official bètas also inserted in the timeline:
> 
> Timestamp   Build ID        Source Stamp  Test Result
> 1377198039  20130822120039  f935e4a1c6dd  ok
> 1377203736  20130822133536  3f1e67b70d5f  ok
> [beta 5]    20130822154523  abf43438122e  ok
> 1377236663  20130822224423  1c39ca9daa6c  affected
> 1377298319  20130823155159  d856284278cc  affected
> [beta 6]    20130826142034  522e9b4cf952  affected
> 1377559364  20130826162244  34fd3a39ab84  affected
> [beta 7]    20130829135643  0422783b9c2f  affected
> 
> Looks like it went wrong on August 22.

So the betas are sometimes a little backdated, since there's a changeset *chosen* to build the beta from.  So https://hg.mozilla.org/releases/mozilla-beta/rev/abf43438122e is actually effectively the same as f935e4a1c6dd (it's the changeset's parent).  So the minimal regression range is a single changeset:
https://hg.mozilla.org/releases/mozilla-beta/rev/1c39ca9daa6c
as you can see (with some reasoning applied) from the pushlog (which is sorted by timestamp and doesn't show ancestry):
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=2ac632b74eb6&tochange=1c39ca9daa6c

Matt, any idea what's up here?

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And thanks for helping find the regression range.

Comment 10

[Mats Palmgren (inactive)]

There's a discussion of a 24b6 regression in that bug and I think Nick is
going to land a fix next week (Matt is on PTO).

Comment 11

[John Daggett (:jtd)]

I'm concerned about this.  If this is really causing a reboot then I think that's a serious concern (as opposed to just causing a crash of Firefox).  Jeroen, are you running on a fully patched version of Windows XP?  Can you verify this on just a single machine or across machines?

Comment 12

[Robert O'Callahan (:roc) (email my personal email if necessary)]

The fix for bug 881634 regression on beta has landed.

It's possible that this bug is due to us trying to allocate too large a surface and Windows XP (or the Nvidia driver) failing to correctly return with an error, instead crashing the system.

Comment 13

[Jeroen van der Gun]

(In reply to John Daggett (:jtd) from comment #11)
My machine (Compaq 8510w laptop with Windows XP Professional) is fully patched (last update automatically installed on August 28). I verified this on a single machine - I also have another Windows XP machine but it's a completely different model. I know a friend who also owns a Compaq 8510w, I may be able to get him to run the same test if you want.

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
Would it be helpful if I test a newer build including this new patch?

Comment 14

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Is the problem fixed in
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-beta-win32/1378082758/ ?

Comment 15

[Jeroen van der Gun]

(In reply to David Baron [:dbaron] (needinfo? me; away Aug 28 - Sep 3) from comment #14)
> Is the problem fixed in
> https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> beta-win32/1378082758/ ?

No. :(

Comment 16

[bhavana bajaj [:bajaj]]

:dbaron setting the assignee as you may have a patch in progress given comment #14? Please reassign if needed.

This is a serious Fx24 regression and may impact sites using Webfonts which may be very common.We are going to build with our second last beta and request to prioritize investigation on this and see if we can have a patch in then by then.

Comment 17

[bhavana bajaj [:bajaj]]

Also needinfo'ing :roc here to get this help given Matt is out.

:roc, I am not sure if :dbaron is the right assignee here or would someone on your team be appropriate, but since this needs urgent urgent attention, requesting your help .

Comment 18

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I don't think we'll be crashing on every site that uses Web fonts or anything like that. I suspect this is going to be limited to resource exhaustion situations on machines with certain drivers. This is still be a problem of course.

Comment 19

[Robert O'Callahan (:roc) (email my personal email if necessary)]

But I don't really have a clue what's going on here.

Jeroen, does your memory usage climb inexorably while you use a canvas demo? See about:memory or Task Manager.

Comment 20

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

comment 14 was a followup to comment 12; I don't have any work in progress.

Bas, could you take this?

Comment 21

[Jeroen van der Gun]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #19)
> Jeroen, does your memory usage climb inexorably while you use a canvas demo?

No. I tested [1] and [2], but in both cases the memory only varies a little around a steady level. [2] ran at about 71 fps with hardware acceleration enabled.

[1] https://developer.mozilla.org/ms/demos/detail/2d-canvas-loader/launch
[2] http://www.smashcat.org/av/canvas_test/

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #18)
> I don't think we'll be crashing on every site that uses Web fonts or
> anything like that. I suspect this is going to be limited to resource
> exhaustion situations on machines with certain drivers.

It doesn't crash on every case of webfonts, but the example URLs in comment 0 indicate the crash is not rare either during normal browsing.

Comment 22

[Benjamin Smedberg]

dmajor, do you have any suggestions for Jeroen about collecting crash information from this computer or at least making BSOD or getting a core instead of an immediate restart?

Comment 23

[(Away)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #22)
> dmajor, do you have any suggestions for Jeroen about collecting crash
> information from this computer or at least making BSOD or getting a core
> instead of an immediate restart?

I would look at the system failure settings (Control Panel, System, Advanced, Startup and Recovery -- though the words may be a little different on XP) and make sure that it is set to collect a kernel memory dump and not automatically restart.

The presence of a kernel debugger should also trigger a break before restarting, but that requires special hardware.

Comment 24

[Milan Sreckovic [:milan] (needinfo for best results)]

George, can you take a look today at Matt's changes and do another review of bug 881634 while we're waiting to see if we can get a reproducible case?

Comment 25

[George Wright (:gw280) (needinfo me!)]

Sure. It'll take me a little while to get an unaccelerated dev machine up but I can look into this today.

Comment 26

[Jeroen van der Gun]

(In reply to David Major [:dmajor] from comment #23)

I disabled automatic restart, but my computer still restarts automatically. Also, writing errors to system log is enabled but I can't find any error or warning in Event Viewer around the time of a crash. Finally, creating a minidump is enabled (including overwriting existing dumps), but the only stored dump I can find is dated over a month ago. Looks like even Windows cannot catch the error.

Comment 27

[(Away)]

(In reply to Jeroen van der Gun from comment #26)
> Looks like even Windows cannot catch the error.

Wow. That's a pretty serious system error. Maybe a triple fault? That makes me suspect the graphics drivers (perhaps provoked by particular call patterns on certain Firefox builds). Is there a difference between the behavior with really old drivers and really new alpha/beta drivers?

Comment 28

[bhavana bajaj [:bajaj]]

Needinfo'ing :dveditz to help with security rating here.

Comment 29

[Jeroen van der Gun]

Attachment: Same info for older Nvidia driver on same machine with same problem

(In reply to David Major [:dmajor] from comment #27)
> Is there a difference between the
> behavior with really old drivers and really new alpha/beta drivers?

For the 7 January 2011 version of the driver, I get exactly the same problem, which is the reason why I updated to the 31 January 2013 version. I've attached exact version details. Note that this is not the original driver version of the machine; I installed the 7 January 2011 version once Firefox started using hardware acceleration (the original driver was too old for that; machine is from 2008). According to the Nvidia site, no newer beta versions exist.

http://www.nvidia.com/Download/index.aspx?lang=en-us
Product Type: Quadro
Product Series: Quadro FX Series (Notebooks)
Product: Quadro FX 570M
Operating System: Windows XP

Comment 30

[George Wright (:gw280) (needinfo me!)]

If you set gfx.canvas.azure.backends to "cairo" does it still crash?

Comment 31

[George Wright (:gw280) (needinfo me!)]

Note for QA: the ThinkPad T61p apparently has this exact graphics card. I'd be surprised if we didn't have a T61p somewhere.

Comment 32

[Jeroen van der Gun]

(In reply to George Wright (:gw280) from comment #30)
> If you set gfx.canvas.azure.backends to "cairo" does it still crash?
Yes, it still crashes. I also tried changing gfx.content.azure.backends, gfx.content.azure.enabled and gfx.font_rendering.cleartype.use_for_downloadable_fonts, but none of these changes prevent the crash. (So far only setting gfx.downloadable_fonts.enabled to false works.)

Comment 33

[George Wright (:gw280) (needinfo me!)]

Needinfoing jfkthame to see if there's any insight he can provide as this appears to be font-related independent of the DrawTarget backend. In the meantime, we are also working on getting test hardware.

Comment 34

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Are the results of those pref checks consistent with this being a regression from https://hg.mozilla.org/releases/mozilla-beta/rev/1c39ca9daa6c ?  Or have we accumulated incorrect data somewhere?

Comment 35

[George Wright (:gw280) (needinfo me!)]

I would find it very surprising if this was caused by that patch, as setting that pref to Cairo should stop DrawTargetSkia from ever being used

Comment 36

[George Wright (:gw280) (needinfo me!)]

That said, I'm working on getting a test build with that patch backed out to double check.

Comment 37

[George Wright (:gw280) (needinfo me!)]

https://tbpl.mozilla.org/?tree=Try&rev=5741c0dc5a0d

Comment 38

[George Wright (:gw280) (needinfo me!)]

Jeroen: when that build is done, can you download it and see if that build crashes in the same way?

Comment 39

[Jeroen van der Gun]

(In reply to George Wright (:gw280) from comment #38)
> Jeroen: when that build is done, can you download it and see if that build
> crashes in the same way?

Seems like I've got your "very surprising" result: this build does not crash. :/

Comment 40

[bhavana bajaj [:bajaj]]

As of today the problem is still concerning for Fx24 because :

a) We are not getting any crash-reports based on the way this crash is encountered
b) It is unknown how many machines or gfx cards this may impact . T61p is one of the popular one that we know has this same card, another one is HP Compaq Mobile Workstation 8510w Notebook 
c) The STR are very simple to hit and it not just crashes firefox but reboots your machine :|
d) Engineering is trying hard but at this point is unable to lay further analysis without having a hardware in hand where this is reproducible 

Benjamin/Goerge/ et al.: Can we try to work around so many unknown's here to get more information ? Is it possible to land something that might avoid the reboot and get us a crash to start with ? That way we may at least have some information at hand ?

Comment 41

[Daniel Veditz [:dveditz]]

(In reply to bhavana bajaj [:bajaj] from comment #28)
> Needinfo'ing :dveditz to help with security rating here.

There's not enough info here, sorry. If it's simply a Denial of Service attack it doesn't rate that high as a "security" issue, though of course taking the machine down is quite severe as DoS bugs go. Maybe "sec-low". On the other hand if it's causing a kernel fault there's a chance it can be used to exploit the OS, not just Firefox, and would be "sec-critical". We're not even getting BSOD information so we just don't know :-(

Comment 42

[Jonathan Kew [:jfkthame]]

(In reply to George Wright (:gw280) from comment #33)
> Needinfoing jfkthame to see if there's any insight he can provide as this
> appears to be font-related independent of the DrawTarget backend.

The connection with fonts is quite mysterious. My only thought at the moment - and it's a very long shot, IMO - is to try transplanting bug 910376 to beta, as it seems possible the problem there might lead us into undefined territory and somehow expose a driver problem if the circumstances are just right.

I've pushed a couple more try jobs to check whether there's any possible connection here. Jeroen, when these builds are ready (it'll be a couple of hours), could you please confirm whether the crash occurs with each of them?

[a] https://tbpl.mozilla.org/?tree=Try&rev=68432e3022e9

[b] https://tbpl.mozilla.org/?tree=Try&rev=b4b3be897979

Thanks for your help and patience as we try to figure out what's happening here!

Comment 43

[Jonathan Kew [:jfkthame]]

(In reply to Jeroen van der Gun from comment #39)
> (In reply to George Wright (:gw280) from comment #38)
> > Jeroen: when that build is done, can you download it and see if that build
> > crashes in the same way?
> 
> Seems like I've got your "very surprising" result: this build does not
> crash. :/

Interesting (and surprising) indeed. However, before we jump to conclusions, we should note that the tryserver build is without PGO, and so may not be directly comparable to the standard builds; could it be that the problem only shows up in PGO builds?

The try jobs I've pushed in comment 42 may help clarify this; if [a] there does NOT crash, then we should suspect PGO as a factor, as that's a simple build of mozilla-beta tip with no changes. If it DOES crash, we know the problem is reproducible even without PGO involved.

Comment 44

[Jeroen van der Gun]

None of the builds from comment 42 cause a crash. Could this be related to the fact that normal Aurora/Nightly builds don't crash either (see comment 4)?

Comment 45

[Jonathan Kew [:jfkthame]]

The try builds from comment 42 are built from mozilla-beta code, not aurora or nightly.

However, if neither of them crash, this suggests that perhaps PGO optimization is part of the problem, and the result from comment 37-39 cannot be trusted as a pointer to the real problem.

I was hoping to create PGO builds of those tryserver jobs, but it's not clear whether that feature is working; no new builds seem to be appearing in tbpl... :(

Comment 46

[Jonathan Kew [:jfkthame]]

OK, I've started a new set of test builds:

[a] https://tbpl.mozilla.org/?tree=Try&rev=1d5703ba4951 (mozilla-beta tip, with PGO)
[b] https://tbpl.mozilla.org/?tree=Try&rev=d53d973e6909 (plus bug 910376)
[c] https://tbpl.mozilla.org/?tree=Try&rev=0905c92e5ff8 (backout 1c39ca9daa6c)

Jeroen: once these are available (in a few hours, if all goes well) please test and let us know whether the crash can be reproduced with any of these.

Comment 47

[Jeroen van der Gun]

(In reply to Jonathan Kew (:jfkthame) from comment #46)

[a] crash
[b] crash
[c] ok

Comment 48

[Jonathan Kew [:jfkthame]]

OK, thanks. So that reconfirms that changeset 1c39ca9daa6c is indeed the one that introduced the crash (though from reading bug 881634, it appears that backing this out would re-introduce other crashes).

However, it also indicates that PGO is necessary to reproduce the crash, as builds [a] and [b] from comment 42, which did NOT crash (comment 44), are built with exactly the same code as [a] and [b] from comment 46; the only difference is that in the latter case, I enabled PGO optimization.

Comment 49

[bhavana bajaj [:bajaj]]

(In reply to Jeroen van der Gun from comment #13)
> (In reply to John Daggett (:jtd) from comment #11)
> My machine (Compaq 8510w laptop with Windows XP Professional) is fully
> patched (last update automatically installed on August 28). I verified this
> on a single machine - I also have another Windows XP machine but it's a
> completely different model. I know a friend who also owns a Compaq 8510w, I
> may be able to get him to run the same test if you want.

That would be really helpful to check and report here. We have been unable to lay our hands on a machine that can reproduce this yet, so it will be good to know if its just one particular hardware where the issue occurs or it is a common scenario.
> 
> (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
> Would it be helpful if I test a newer build including this new patch?

Comment 50

[bhavana bajaj [:bajaj]]

:gw280/:roc/:bas,

This was discussed in today's crashkill and since its proved that a backout of cset that landed in beta for https://bugzilla.mozilla.org/show_bug.cgi?id=881634, we discussed that may be the best solution for Fx24. Benjamin recommended to follow-up with with you all that the backout would be safe and not leave any possible security hole.

Comment 51

[bhavana bajaj [:bajaj]]

(In reply to bhavana bajaj [:bajaj] from comment #50)
> :gw280/:roc/:bas,
> 
> This was discussed in today's crashkill and since its proved that a backout
> of cset that landed in beta for
> https://bugzilla.mozilla.org/show_bug.cgi?id=881634, we discussed that may
> be the best solution for Fx24. Benjamin recommended to follow-up with with
> you all that the backout would be safe and not leave any possible security
> hole.

redirecting my question to :mattwoodrow as George said he is back from PTO and since he is landed the patch in 881634.

Comment 52

[bhavana bajaj [:bajaj]]

Requesting feedback from support to see if we are seeing any reports on this issue.

Comment 53

[Tyler Downer [He/Him]]

So I took a look into this, and while there is general feedback around crashes on Beta nobody is explicitly saying "restart, reboot, windows closed," etc. which are things I'd expect to see if users were experiencing this.I also looked on our Forums and failed to see a single piece of feedback on a crash that was similar to this issue. So it's safe to assume that our beta population isn't seeing much if any of this problem.

While shipping with this issue obviously isn't ideal I'd rather leave the fix for the top crasher in than pull that out (because that was something we saw feedback around). Since there is a workaround we can write an article if this explodes (obviously fixing this asap is awesome

Thanks!

Comment 54

[bhavana bajaj [:bajaj]]

(In reply to Tyler Downer [:Tyler] from comment #53)
> So I took a look into this, and while there is general feedback around
> crashes on Beta nobody is explicitly saying "restart, reboot, windows
> closed," etc. which are things I'd expect to see if users were experiencing
> this.I also looked on our Forums and failed to see a single piece of
> feedback on a crash that was similar to this issue. So it's safe to assume
> that our beta population isn't seeing much if any of this problem.
> 
> While shipping with this issue obviously isn't ideal I'd rather leave the
> fix for the top crasher in than pull that out (because that was something we
> saw feedback around)

Unfortunately, https://bugzilla.mozilla.org/show_bug.cgi?id=881634#c44 is not fixed. :tracy is digging further here to see why crash signature has disappeared from b8 topcrashes at least.

> Since there is a workaround we can write an article if
> this explodes (obviously fixing this asap is awesome
> 
> Thanks!

Comment 55

[Matt Woodrow (:mattwoodrow)]

Since this crash only occurs on PGO builds, it seems very likely that it is caused by a bug in the PGO compiler. Possibly also requiring some interaction with a specific graphics card.

If we can determine which cpp file is likely to be causing the issue, and disable PGO for just that file, we might be able to work around this.

Jonathon: Do you have any idea which file(s) is(/are) the likely culprit(s) (given that it requires the pref gfx.downloadable_fonts.enabled to be true)?

We're probably a bit late now for this release, but if we could get a try push with PGO enabled for everything except that file today we might be able to land something tomorrow.

I'm going ahead with a backout of the regressing changeset now, since we need to get back to a known good state before we can go to RC.

Comment 56

[Matt Woodrow (:mattwoodrow)]

For disabling PGO, I think we need to something like this around the offending code (if we can figure out what that is): http://mxr.mozilla.org/mozilla-central/source/js/src/vm/Stack.cpp#511

Comment 57

[bhavana bajaj [:bajaj]]

Needinfo'ing Jereon, to help with request in comment# 49 .

Comment 58

[(Away)]

I'm hesitant to call it a PGO compiler bug. I suspect the real trigger for the crash is being in some particular state, or with some particular timing patterns. If disabling PGO changes anything, it's probably because doing so scrambles the execution flow throughout the binary, and all the states and timings change. I think the same thing is going on in comment 35 where the regressing changeset appears to be code that isn't even executed.

What I am trying to say is that "no crash without PGO" plus "no crash without downloadable_fonts" does not necessarily equal "no crash without PGO downloadable_fonts".

That said, it's worth a try. If there is some obvious file that comes to mind, and it stops the crash, then great. But I wouldn't spend too much time searching, since it might not be something that can be isolated to a single file.

Comment 59

[Jeroen van der Gun]

Attachment: No crash on this Windows 7 machine

(In reply to bhavana bajaj [:bajaj] from comment #49)

It turns out my friend upgraded to Windows 7. His machine does have the same graphics card, but he cannot reproduce the crash. For completeness I attached his about:support.

Comment 60

[Jonathan Kew [:jfkthame]]

(In reply to Matt Woodrow (:mattwoodrow) from comment #55)
> Jonathon: Do you have any idea which file(s) is(/are) the likely culprit(s)
> (given that it requires the pref gfx.downloadable_fonts.enabled to be true)?

Downloadable fonts are handled primarily by gfxUserFontSet.cpp and gfxGDIFontList.cpp (as this is on WinXP).

However, given that the change that triggered the crashing behavior doesn't appear to have anything to do with fonts, I suspect it may not really be a bug in font-related code as such (or in the PGO compilation of that code). To completely crash/reboot Windows in this way surely requires an underlying driver-level or Windows kernel-level bug; it just happens that we've managed to hit exactly the right circumstances to tickle it.

I suspect turning off PGO fixes the issue not because of an actual bug in the PGO compilation itself, but because control flow, timing, memory layout, etc., change sufficiently that we no longer happen to hit the magic combination of factors that lead to the crash. Quite likely -any- substantial rearrangement of the code (e.g. rearranging the link order of libraries, or changing the default sizes of various allocated structures, or who-knows-what) could lead to a similar "fix".

But anyhow, I've pushed a new set of PGO try builds, to see if any of these happen to resolve the problem. (Though I have no great confidence in this!) It's equally possible that the key place to touch would be somewhere in gfx/2d drawtarget or surface code, but I don't have any idea where to poke at that.

mozilla-beta before backout of 881634, for reference (expected to crash):
[0] https://tbpl.mozilla.org/?tree=Try&rev=e9362d515ad8

PGO disabled in gfxGDIFont.cpp:
[1] https://tbpl.mozilla.org/?tree=Try&rev=61d9295492c2

PGO disabled in gfxGDIFontList.cpp:
[2] https://tbpl.mozilla.org/?tree=Try&rev=9bab8347086f

PGO disabled in gfxUserFontSet.cpp:
[3] https://tbpl.mozilla.org/?tree=Try&rev=e4db81763c27

Jeroen, once these are ready, please do yet another round of testing and let us know which of them do/don't crash for you. Thanks!

Comment 61

[bhavana bajaj [:bajaj]]

In addition, our latest Fx beta (24.0b10) has the backout of 881634, which should fix the issue here.Will be great if you can confirm as that should be the final build expected to hit our release audience.

Comment 62

[Jeroen van der Gun]

All four builds from comment 60 are still affected. However, since I now did the tests using Firefox health report rather than using geeksphone.com, I did observe some interesting things. I tested each build three times.

* Once, build [1] showed the basic layout of the health report, without any text filled in, before crashing. Once, build [3] did the same.

* Once, build [2] showed the health report with only the stats and graph missing, but the text filled in, including webfont text, before crashing.

* Once, build [3] showed a blank page instead of the health report, but did not crash. When I loaded geeksphone.com in another tab, it crashed after all.

Because of this, I retested 24.0b6 three times with the Firefox health report, and in one case the health report showed up correctly without crashing.

I think this reflects the fact this is not clear boolean logic, but more fuzzy, as already indicated in comment 0. Perhaps geeksphone.com is a more reliable method to crash the browser due to the graphic animations in addition to webfonts (although no animations are visible on the screen prior to the crash). I did all earlier tests with geeksphone.com and afaict there were no contradictory results.

It does suggest that comment 59 may need further verification, as it was a single attempt to crash with the health report only.

Meanwhile, 24.0b10 is ok, I tried to crash it twice with both Firefox health report and geeksphone.com, but it works correctly.

Comment 63

[Jeroen van der Gun]

Regarding comment 59, also no crash is observed using a combination of geeksphone.com and the health report. This verifies the conclusion that that Windows 7 machine is unaffected.

Comment 64

[bhavana bajaj [:bajaj]]

needinfo'ing :mschifer here to see if was able to reproduce the issue on 24.0b6 in house with the hardware we recently got.

Comment 65

[Marc Schifer [mschifer]]

I was unable to reproduce the bug in the provided lap top in either Beta 6 or Beta 10

Comment 66

[u279076]

Can this bug be marked WONTFIX given Firefox 24 is now behind us and Firefox 25 is apparently unaffected?

Comment 67

[Mats Palmgren (inactive)]

Jeroen, please reopen this bug if you can reproduce it in a recent version.