Last Comment Bug 716232 - SIGSEGV (segmentation violation) crash in Javascript at ChatZilla startup
: SIGSEGV (segmentation violation) crash in Javascript at ChatZilla startup
Status: RESOLVED FIXED
: crash, regression, relnote
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: 12 Branch
: All Linux
: -- critical with 1 vote (vote)
: ---
Assigned To: David Mandelin [:dmandelin]
:
: Jason Orendorff [:jorendorff]
Mentors:
Depends on:
Blocks: 703157
  Show dependency treegraph
 
Reported: 2012-01-07 07:03 PST by Tony Mechelynck [:tonymec]
Modified: 2012-06-01 09:07 PDT (History)
14 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
unaffected
unaffected
+
verified
+
verified
unaffected
affected


Attachments
ChatZilla prefs, see comment #11 (6.50 KB, text/plain)
2012-01-09 18:04 PST, Tony Mechelynck [:tonymec]
no flags Details
custom networks.txt (3.24 KB, text/plain)
2012-01-09 18:05 PST, Tony Mechelynck [:tonymec]
no flags Details
stdout/stderr from try-build #1 (68 bytes, text/plain)
2012-02-06 21:34 PST, Tony Mechelynck [:tonymec]
no flags Details
log in Feb-29 SeaMonkey, starting at cZ startup (9.18 KB, text/plain)
2012-02-28 21:52 PST, Tony Mechelynck [:tonymec]
no flags Details

Description Tony Mechelynck [:tonymec] 2012-01-07 07:03:53 PST
This bug was filed from the Socorro interface and is 
report bp-591081ef-71ca-4ef0-af68-c95b72120107 .
============================================================= 
Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120107 Firefox/12.0a1 SeaMonkey/2.9a1 ID:20120107003001
Comment 1 Tony Mechelynck [:tonymec] 2012-01-07 07:16:14 PST
oops, hit Enter too fast.

I had this crash twice in succession at startup, after opening all three of:
- ChatZilla
- MailNews 3-pane
- Browser about:blank
but while still auto-connecting and auto-joining on IRC, and getting POP3 mail at startup.
Then I tried starting only the browser, and it didn't crash.
bp-591081ef-71ca-4ef0-af68-c95b72120107
bp-1d41888f-106e-44b3-b302-d23db2120107

This (at first and second startup after installing this nightly) is the first time I see this crash: on yesterday's nightly I didn't experience it.

Of the three "possibly related" bugs mentioned by Soccorro, bug 655561 is a dupe, bug 655562 is FIXED (which is obviously not the case of this one) and bug 654903 (reported for Firefox 6) has a stack which (to me) looks very different of this one.

Here comes the crash data from Soccorro for the first of the two crashes mentioned above:

Signature 	js::gc::PushMarkStack More Reports Search
UUID	591081ef-71ca-4ef0-af68-c95b72120107
Date Processed	2012-01-07 06:56:36.352604
Uptime	41
Last Crash	1.7 days before submission
Install Age	41 seconds since version was first installed.
Install Time	2012-01-07 14:52:58
Product	SeaMonkey
Version	2.9a1
Build ID	20120107003001
Release Channel	nightly
OS	Linux
OS Version	0.0.0 Linux 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64
Build Architecture	amd64
Build Architecture Info	family 15 model 4 stepping 1
Crash Reason	SIGSEGV
Crash Address	0xfc0b8
User Comments	at startup. All windows (ChatZilla, MailNews, Browser) already open. Still busy with cZ start-up auto-connects and auto-joins, and with getting POP3 mail and RSS feeds at startup.
App Notes 	

GLXtest process failed (exited with status 1): X error occurred in GLX probe, error_code=9, request_code=55, minor_code=0



Processor Notes 	
EMCheckCompatibility	False
Winsock LSP	

Adapter Vendor ID	
Adapter Device ID	
Bugzilla - Report this Crash
Related Bugs

    655561 RESOLVED DUPLICATE Firefox 6.0a1 Crash Report [@ js::gc::PushMarkStack ]
    655562 RESOLVED FIXED Crash in the No Comply demo (mozjs!js::GCMarker::drainMarkStack+0x0000029b) [@ js::gc::PushMarkStack ][@ mozjs.dll]
    654903 NEW Firefox 6.0a1 Crash Report [@ js::gc::PushMarkStack ]

Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	libxul.so 	js::gc::PushMarkStack 	js/src/jsgc.h:641
1 	libxul.so 	js::GCMarker::processMarkStackTop 	js/src/jsgcmark.cpp:1122
2 	libxul.so 	js::GCMarker::drainMarkStack 	js/src/jsgcmark.cpp:1164
3 	libxul.so 	GCCycle 	js/src/jsgc.cpp:2706
4 	libxul.so 	js_GC 	js/src/jsgc.cpp:3163
5 	libxul.so 	nsXPConnect::Collect 	js/xpconnect/src/nsXPConnect.cpp:412
6 	libxul.so 	nsXPConnect::GarbageCollect 	js/xpconnect/src/nsXPConnect.cpp:420
7 	libxul.so 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:428
8 	libxul.so 	nsTimerEvent::Run 	xpcom/threads/nsTimerImpl.cpp:524
9 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
10 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
11 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
12 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
13 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
14 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
15 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
16 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
17 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
18 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
19 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
20 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
21 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
22 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
23 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
24 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
25 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
26 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
27 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
28 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
29 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
30 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
31 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
32 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
33 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
34 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
35 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
36 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
37 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
38 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
39 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
40 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
41 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
42 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
43 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
44 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
45 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
46 	libxul.so 	nsThread::Shutdown 	xpcom/threads/nsThread.cpp:502
47 	libxul.so 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:195
48 	libxul.so 	nsProxyObjectCallInfo::Run 	xpcom/proxy/src/nsProxyEvent.cpp:182
49 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:660
50 	libxul.so 	NS_ProcessNextEvent_P 	objdir/mozilla/xpcom/build/nsThreadUtils.cpp:245
51 	libxul.so 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:110
52 	libxul.so 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:201
53 	libxul.so 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:189
54 	libxul.so 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:220
55 	libxul.so 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3537
56 	seamonkey-bin 	main 	suite/app/nsSuiteApp.cpp:103
57 	libc-2.14.1.so 	libc-2.14.1.so@0x2123c 	
58 	seamonkey-bin 	Output 	suite/app/nsSuiteApp.cpp:72
Comment 2 Tony Mechelynck [:tonymec] 2012-01-07 11:55:05 PST
Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120107 Firefox/12.0a1 SeaMonkey/2.9a1 ID:20120107003001

bp-84711aaa-7e56-4d5d-b57e-a65152120107
bp-495bede4-44a9-4c55-96fe-bc94b2120107

This last one is interesting: instead of opening all three windows at startup, I opened them one by one: started the browser on about:blank (no crash), started the mailer and let it poll all servers until it had done with it (still no crash) then I clicked the cZ button on the status bar, the ChatZilla window opened, with client tab and my three server tabs, but then almost immediately, before the end of my auto-joins, there was a crash. So I changed general.startup.chat from true to false in prefs.js, started SeaMonkey normally, and the browser and mailer (but of course not the chat) came up, with no crash.

My ChatZilla version is 0.9.88.
Comment 3 Tony Mechelynck [:tonymec] 2012-01-07 12:03:42 PST
P.S. Also, now that I look at it, this last crash is different: js::Shape::initDictionaryShape instead of js::gc::PushMarkStack. Still a js-related crash but the stack looks different. Shall I move that last crash to a new bug or add its signature on top of this bug?
Comment 4 Robert Kaiser 2012-01-08 06:33:22 PST
PushMarkStack and AFAIK also many js::Shape functions are being called in the GC marking phase. I wonder why GC would mark objects on startup, but apart from that, those signatures are pretty common and hard to impossible to diagnose.
Comment 5 Tony Mechelynck [:tonymec] 2012-01-08 09:15:10 PST
Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120108 Firefox/12.0a1 SeaMonkey/2.9a1 ID:20120108003001

bp-74ff4128-f8a2-4029-a4b8-05e2f2120108

at ChatZilla startup, browser already started, mailer not started. Again related but different: [@ js::gc::Arena::finalize<JSObject>]

I have a feeling that these crashes, all at cZ startup and all in js:: something, are somehow related so I'm bringing them all here despite the non-identical signatures and stacks.

This one is a SIGSEGV at 0x0: dereferencing a null pointer? The one in comment #1 is a SIGSEGV but not at 0x0: an invalid pointer? (use-after-free?) Let's see the others:
bp-1d41888f-106e-44b3-b302-d23db2120107 — SIGSEGV at 0xfc0b8 (the same address as the first)
bp-84711aaa-7e56-4d5d-b57e-a65152120107 — ditto
bp-495bede4-44a9-4c55-96fe-bc94b2120107 — SIGSEGV at 0x0

This crash is solid (for me), the single STR is:
1. Start ChatZilla
so I feel confident that yesterday's build (see comment #0) was the "first bad nightly".

Which commits happened on 2012-01-06? On comm-central only one (bug 715793) but which might have far-reaching consequences on dep builds
On mozilla-central (including the first half-hout after midnight on the 7) what looks suspicious to me?
Bug 715852 about ecma-something
Bug 716139 about JS but probably irrelevant
Bug 707321 ditto
Bug 715883 about eliminating stuff "no longer used" in JS garbage collector

That's what looks "most relevant" to me but don't take my word for it, check the hg.m.o logs yourself if you see nothing obvious in the patches for the above.
Comment 6 Robert Kaiser 2012-01-08 16:42:11 PST
Does setting this to status-firefox12:affected mean that you can reproduce by installing ChatZilla on FF Nightly? How about Aurora? Can you find a regression range on Firefox?
Comment 7 Tony Mechelynck [:tonymec] 2012-01-08 17:07:31 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #6)
> Does setting this to status-firefox12:affected mean that you can reproduce
> by installing ChatZilla on FF Nightly? How about Aurora? Can you find a
> regression range on Firefox?

No, it means that the seamonkey-2.9:affected flag has been erased as a result of moving this bug to Core (where that flag is not available), and that "Core::Javascript" behaviour of SeaMonkey 2.9a1 is supposed to be identical to that of Firefox 12.0a1. I would have set Gecko-12:affected if that had been available, but it isn't. <span class="irony">Maybe the Gecko developers or the people customizing Bugzilla for them should be shown the http://geckoisgecko.org site to remind them that Gecko is not only Firefox. :-/ </span>

I never had this bug before yesterday (which is significantly after the latest Aurora merge) and I've had it at every ChatZilla startup since then. I haven't tested an actual Nightly nightly, "only" SeaMonkey trunk nightlies.
Comment 8 Tony Mechelynck [:tonymec] 2012-01-08 18:13:59 PST
bp-77cbea6c-23db-4326-803c-604d32120108 js::gc::Arena::finalize<JSObject> 

Mailer and browser opened at startup (no crash) then manually ChatZilla (crash). All three server tabs and many (auto-started) channel and query tabs appeared in the few seconds before the crash.

I had disabled the recently updated Nightly Tester Tools extension to make sure that it didn't contribute to the crash, and it doesn't.
Comment 9 Tony Mechelynck [:tonymec] 2012-01-08 19:14:03 PST
Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120108 Firefox/12.0a1
Build ID: 20120108031024

bp-2952ee3e-7856-4bd5-b32e-2bcf42120108 js::gc::FinalizeArenas 

@KaiRo: After copying the chatzilla/ subfolder and the prefs.js lines about extensions.irc.* to my Firefox testing profile, and installing in it /usr/local/seamonkey/distribution/extensions/{59c81df5-4b7a-477b-912d-4e0fdf64e5f2}.xpi (the ChatZilla version distributed with today's SeaMonkey nightly, which I installed by browsing to its directory in Nightly and clicking the .xpi name) the crash happened also in this Nightly nightly, so now I can tell you: yes, I can reproduce the crash every time in Firefox 12.0a1

The crash does not happen in a "virgin" profile where no auto-connects or auto-joins are preset for ChatZilla.
Comment 10 Tony Mechelynck [:tonymec] 2012-01-09 17:32:38 PST
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.26pre) Gecko/20120107 Namoroka/3.6.26pre - Build ID: 20120107033207

(as expected) the bug is not present in this build. Of course this is not the narrowest possible regression range ;-) but I happened to have this Firefox build lying idle on this computer.
Comment 11 Tony Mechelynck [:tonymec] 2012-01-09 18:04:25 PST
Created attachment 587220 [details]
ChatZilla prefs, see comment #11

Sensitive information (such as nicknames, usernames, /nickserv identify, /msg usersrv login, etc.) has been removed; you should of course add it back if you want to try to reproduce the bug with these settings.

I have a custom networks.txt (including connection data for one of my three auto-connect networks, not included in ChatZilla's default list); I'll attach it next.
Comment 12 Tony Mechelynck [:tonymec] 2012-01-09 18:05:47 PST
Created attachment 587221 [details]
custom networks.txt
Comment 13 Tony Mechelynck [:tonymec] 2012-01-09 18:17:34 PST
P.S. This networks.txt is intentionally biased in favour of European servers and in particular servers "not too far from where I live" (Brussels, Belgium). I don't think it would make much of a difference in bug reproducibility if you optimize it for where _you_ live.
Comment 14 Tony Mechelynck [:tonymec] 2012-01-27 20:25:10 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4)
> PushMarkStack and AFAIK also many js::Shape functions are being called in
> the GC marking phase. I wonder why GC would mark objects on startup, but
> apart from that, those signatures are pretty common and hard to impossible
> to diagnose.

AFAICT this crash appeared about when bug 715883 (which removed something supposedly "unused" in the JS GC) was "fixed". Shouldn't that be investigated?
Comment 15 Robert Kaiser 2012-01-28 05:30:28 PST
(In reply to Tony Mechelynck [:tonymec] from comment #14)
> AFAICT this crash appeared about when bug 715883 (which removed something
> supposedly "unused" in the JS GC) was "fixed". Shouldn't that be
> investigated?

Yes, _that_ should be investigated. Should be easy enough to try a build with a backout of that, or try to zero in on the changeset by trying build exactly before and after this landing (if you can't build yourself, someone can whip up try builds for you as needed). But the signatures themselves are not too helpful.
Comment 16 Tony Mechelynck [:tonymec] 2012-01-28 16:11:01 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #15)
> (In reply to Tony Mechelynck [:tonymec] from comment #14)
> > AFAICT this crash appeared about when bug 715883 (which removed something
> > supposedly "unused" in the JS GC) was "fixed". Shouldn't that be
> > investigated?
> 
> Yes, _that_ should be investigated. Should be easy enough to try a build
> with a backout of that, or try to zero in on the changeset by trying build
> exactly before and after this landing (if you can't build yourself, someone
> can whip up try builds for you as needed). But the signatures themselves are
> not too helpful.

no, ATM I can't build myself; if you (or someone) could, as you say, "whip up a try build" (or two or three) for linux-x86_64, I could test that and label the builds as "good" or "bad"; since I get the crash every time I ought not to have to say "maybe".
Comment 17 David Mandelin [:dmandelin] 2012-01-30 12:11:35 PST
(In reply to Tony Mechelynck [:tonymec] from comment #16)
> (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #15)
> > (In reply to Tony Mechelynck [:tonymec] from comment #14)
> > > AFAICT this crash appeared about when bug 715883 (which removed something
> > > supposedly "unused" in the JS GC) was "fixed". Shouldn't that be
> > > investigated?
> > 
> > Yes, _that_ should be investigated. Should be easy enough to try a build
> > with a backout of that, or try to zero in on the changeset by trying build
> > exactly before and after this landing (if you can't build yourself, someone
> > can whip up try builds for you as needed). But the signatures themselves are
> > not too helpful.
> 
> no, ATM I can't build myself; if you (or someone) could, as you say, "whip
> up a try build" (or two or three) for linux-x86_64, I could test that and
> label the builds as "good" or "bad"; since I get the crash every time I
> ought not to have to say "maybe".

Are you using the standalone Chatzilla or the Firefox add-on?
Comment 18 Robert Kaiser 2012-01-30 12:27:36 PST
(In reply to David Mandelin from comment #17)
> Are you using the standalone Chatzilla or the Firefox add-on?

The crash reports mentioned here are all from the add-on, either in SeaMonkey or Firefox, see bp-2952ee3e-7856-4bd5-b32e-2bcf42120108 for the latter.

What he'd need are Firefox try builds to try with to narrow down the regression range.

Tony, you already narrowed down to a regression day, right?
Comment 19 Tony Mechelynck [:tonymec] 2012-01-30 16:23:02 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #18)
[...]
> Tony, you already narrowed down to a regression day, right?

Yes, see comment #1: the nightly from 7 January was "first bad", the one from 6 January was "last good". Typically the source for SeaMonkey 2.9a1 linux-x86_64 is pulled around 0h30 Mountain View time every night, so the regression appeared on 6 January (or within half an hour after midnight on 7 January).

See comment #5 about the "detective work" I did, trying to find what might have caused the regression.
Comment 20 David Mandelin [:dmandelin] 2012-02-03 14:36:03 PST
(In reply to Tony Mechelynck [:tonymec] from comment #19)
> (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #18)
> [...]
> > Tony, you already narrowed down to a regression day, right?
> 
> Yes, see comment #1: the nightly from 7 January was "first bad", the one
> from 6 January was "last good". Typically the source for SeaMonkey 2.9a1
> linux-x86_64 is pulled around 0h30 Mountain View time every night, so the
> regression appeared on 6 January (or within half an hour after midnight on 7
> January).
> 
> See comment #5 about the "detective work" I did, trying to find what might
> have caused the regression.

Sorry about the delay--I've been out sick the past few days. Anyway, I'm ready to create some try builds for you to test, but a couple of questions:

1. Looking at comment 9 and comment 10, it looks like for Firefox, the nightly from 7 Jan is good, and 8 Jan is the first bad. Is that correct?

2. What platform? linux64? Any others?
Comment 21 Tony Mechelynck [:tonymec] 2012-02-03 18:30:12 PST
(In reply to David Mandelin from comment #20)
> (In reply to Tony Mechelynck [:tonymec] from comment #19)
> > (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #18)
> > [...]
> > > Tony, you already narrowed down to a regression day, right?
> > 
> > Yes, see comment #1: the nightly from 7 January was "first bad", the one
> > from 6 January was "last good". Typically the source for SeaMonkey 2.9a1
> > linux-x86_64 is pulled around 0h30 Mountain View time every night, so the
> > regression appeared on 6 January (or within half an hour after midnight on 7
> > January).
> > 
> > See comment #5 about the "detective work" I did, trying to find what might
> > have caused the regression.
> 
> Sorry about the delay--I've been out sick the past few days. Anyway, I'm
> ready to create some try builds for you to test, but a couple of questions:
> 
> 1. Looking at comment 9 and comment 10, it looks like for Firefox, the
> nightly from 7 Jan is good, and 8 Jan is the first bad. Is that correct?

I'm not 100% sure about Firefox: 8 Jan is bad but not necessarily first bad. For SeaMonkey, 6 Jan is last good, 7 Jan is fist bad and both were built from source pulled around 0h30 PST. Maybe build a few test builds before and after the fix for bug 715883; possibly also before and after the other bugs mentioned in comment #5 which were all "fixed" on January 6 (the assignee for bug 715883 doesn't believe that his bug "could" cause this crash, see bug 715883 comment #6).

> 
> 2. What platform? linux64? Any others?

Linux64. I expect the bug to be platform-agnostic, and my only box has a linux64 OS (openSUSE 12.1 x86_64). All my crashes happened on linux64, I could test linux32-on-linux64 but didn't yet, and I don't think it's worth the trouble to test them both.

Note: I'll be at FOSDEM (in my hometown) with no computer most of this weekend: don't expect me to give answers before Monday afternoon CET (UTC+1).
Comment 22 David Mandelin [:dmandelin] 2012-02-06 16:37:21 PST
(In reply to Tony Mechelynck [:tonymec] from comment #21)
> (In reply to David Mandelin from comment #20)
> > (In reply to Tony Mechelynck [:tonymec] from comment #19)
> > > (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #18)
> > 1. Looking at comment 9 and comment 10, it looks like for Firefox, the
> > nightly from 7 Jan is good, and 8 Jan is the first bad. Is that correct?
> 
> I'm not 100% sure about Firefox: 8 Jan is bad but not necessarily first bad.
> For SeaMonkey, 6 Jan is last good, 7 Jan is fist bad and both were built
> from source pulled around 0h30 PST. Maybe build a few test builds before and
> after the fix for bug 715883; possibly also before and after the other bugs
> mentioned in comment #5 which were all "fixed" on January 6 (the assignee
> for bug 715883 doesn't believe that his bug "could" cause this crash, see
> bug 715883 comment #6).
> 
> > 
> > 2. What platform? linux64? Any others?
> 
> Linux64. I expect the bug to be platform-agnostic, and my only box has a
> linux64 OS (openSUSE 12.1 x86_64). All my crashes happened on linux64, I
> could test linux32-on-linux64 but didn't yet, and I don't think it's worth
> the trouble to test them both.

Thanks for the info. Here are two try builds of Firefox from between Jan 7 and Jan 8 that seem to cut off a few chunks of JS-related landings:

  #1 (f0eab7fd20af): https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-d1227a68750b/try-linux64/

  #2 (48928463f978): https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-77bf0f19c803/try-linux64/

Could you please test these and tell us whether it works or not in each one?
Comment 23 Tony Mechelynck [:tonymec] 2012-02-06 21:34:21 PST
Created attachment 594911 [details]
stdout/stderr from try-build #1

#1 is bad, bp-783c0706-8a32-49d6-b3c5-4ebca2120207 with the commandline

./firefox/firefox -no-remote -P testing -chat > firefox.log 2>&1 &

(which starts only ChatZilla). The CZ window is seen to open, many channels are /joined (auto-joined at cZ startup), then the cZ window disappears and the Breakpad popup opens. Soccorro doesn't know the symbols for this build (or maybe I should have installed them but didn't know how).
Comment 24 Tony Mechelynck [:tonymec] 2012-02-06 21:43:43 PST
(In reply to Tony Mechelynck [:tonymec] from comment #23)
P.S. Associated .txt file:

20120206134858
http://hg.mozilla.org/try/rev/d1227a68750b
Comment 25 Tony Mechelynck [:tonymec] 2012-02-06 21:55:38 PST
20120206134629
http://hg.mozilla.org/try/rev/77bf0f19c803

#2 is bad: bp-bc025b6e-1c10-4f8b-9dc0-52fd92120207 (same symptoms as in comment #23).

I expect the critical "bad changeset" to have been committed on 6 (not 7) January. Reminder: The "first bad" build was built on 7 January from source pulled from mozilla-central (and comm-central etc.) at approx. half past midnight Mountain View Time. The "last good" build was about 24h earlier (i.e. on 6 January shortly after midnight).
Comment 26 Tony Mechelynck [:tonymec] 2012-02-06 22:13:40 PST
@dmandelin: I'd be willing, if possible, to test a try-build consisting of the latest mozilla-central source with the fix for bug 715883 rolled back: if that is good, it would vindicate my hypothesis from comment #5 where that bug seemed "most likely" to me (who know nothing about the JS engine) as a possible culprit. Conversely, if such a build is "bad" bug 715883 is shown to be irrelevant and the culprit is to be looked for elsewhere.
Comment 27 David Mandelin [:dmandelin] 2012-02-07 11:48:18 PST
(In reply to Tony Mechelynck [:tonymec] from comment #26)
> @dmandelin: I'd be willing, if possible, to test a try-build consisting of
> the latest mozilla-central source with the fix for bug 715883 rolled back:
> if that is good, it would vindicate my hypothesis from comment #5 where that
> bug seemed "most likely" to me (who know nothing about the JS engine) as a
> possible culprit. Conversely, if such a build is "bad" bug 715883 is shown
> to be irrelevant and the culprit is to be looked for elsewhere.

I took build #1 from a point before bug 715883, so I think we've already ruled it out. Based on your suspicion of a Jan 6 landing, I think bug 712714 is the likeliest. I'm setting up a few more try builds now, they should be ready in a few hours.
Comment 29 Tony Mechelynck [:tonymec] 2012-02-08 03:12:03 PST
20120207114313
http://hg.mozilla.org/try/rev/68eb2f6cd67d
#3 is Bad, bp-aec3725b-ef78-4085-8dc6-a746c2120208

20120207115041
http://hg.mozilla.org/try/rev/2e27de92b0dc
#4 is Bad, bp-66d4d4f8-badc-4f2c-adf5-1fd452120208

20120207115107
http://hg.mozilla.org/try/rev/5414e8a3712c
#5 is Bad, bp-f38bd925-a0cf-400e-bb4a-bf8292120208

I'd have tried to test a Firefox nightly in addition to the above, but Nightlies from January 1 to 7 seem to be lost (not archived, or not built).
Comment 30 David Mandelin [:dmandelin] 2012-02-08 16:43:14 PST
(In reply to Tony Mechelynck [:tonymec] from comment #29)
> I'd have tried to test a Firefox nightly in addition to the above, but
> Nightlies from January 1 to 7 seem to be lost (not archived, or not built).

Surprising that they're all bad. It's nice that it wasn't bug 712714, though. I don't know what happened to Jan 1-7 nightlies either--I also found them missing while trying to bisect something recently. Thanks for doing all this testing, btw!

And I have more test builds, from earlier on Jan 6:

#6: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-4f6b3f057931/
#7: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-a7e4585aeb39/
#8: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-9fbae28df2e5/
Comment 31 Tony Mechelynck [:tonymec] 2012-02-08 18:41:30 PST
(In reply to David Mandelin from comment #30)
> [...] Thanks for doing all this testing, btw!
> [...]

It's only natural: I want to be able to use cZ again in my usual (trunk) SeaMonkey rather than in an ad-hoc build of Fx 3.6 which would be obsolete if it weren't supported as ESR.

20120208110435
http://hg.mozilla.org/try/rev/4f6b3f057931
#6 is Good
cZ starts up, auto-joins a number of channels, and is seen by my Konversation (KDE) IRC client. No crash.

20120208110856
http://hg.mozilla.org/try/rev/a7e4585aeb39
#7 is Good. Same symptoms.

20120208110635
http://hg.mozilla.org/try/rev/9fbae28df2e5
#8 is Good.
Comment 32 David Mandelin [:dmandelin] 2012-02-08 19:08:26 PST
OK, we're down to this range:

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8062086f8e00&tochange=57a33c9f08b1

which has only 3 JS engine changes. Bug 703157 looks the most likely. Next tests:

#9:  https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-0b314c170e1a/
#10: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dmandelin@mozilla.com-ec9562cdd5a8/

They're still building, they should be done in an hour or two.
Comment 33 Tony Mechelynck [:tonymec] 2012-02-08 21:34:30 PST
20120208191015
http://hg.mozilla.org/try/rev/0b314c170e1a
#9 is Good.

20120208190535
http://hg.mozilla.org/try/rev/ec9562cdd5a8
#10 is Bad: bp-b3ff546b-9fe8-4f22-b67c-601792120209
Comment 34 David Mandelin [:dmandelin] 2012-02-09 11:17:12 PST
The regressing changeset is:

changeset:   83883:faf5f8842fec
user:        Brian Hackett <bhackett1024@gmail.com>
date:        Thu Jan 05 06:38:46 2012 -0800
summary:     Don't modify dictionary shapes in place, bug 703157. r=luke
Comment 35 Tony Mechelynck [:tonymec] 2012-02-21 06:01:28 PST
@bhackett, @luke: ping (see comment #34): I get the crash every time, I'm willing to test any try-build, but I'm not up to coding.
Comment 36 Brian Hackett (:bhackett) 2012-02-21 07:40:27 PST
Can you try these builds (when they appear):

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bhackett@mozilla.com-9b408f8a057b

This just backs out the patch.  I don't see anything wrong with the patch so without some solid STR a backout is really the only option.
Comment 37 Tony Mechelynck [:tonymec] 2012-02-21 08:42:19 PST
(In reply to Brian Hackett (:bhackett) from comment #36)
> Can you try these builds (when they appear):
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bhackett@mozilla.
> com-9b408f8a057b
> 
> This just backs out the patch.  I don't see anything wrong with the patch so
> without some solid STR a backout is really the only option.

This indeed does not crash for me.
Comment 38 Tony Mechelynck [:tonymec] 2012-02-21 08:45:12 PST
P.S. My STR are very simple: "1. Start ChatZilla". This crashes every time for me. I've tried to attach as much info as I could think of about my cZ config in the hope of making the problem reproducible.
Comment 39 Tony Mechelynck [:tonymec] 2012-02-21 09:01:19 PST
P.P.S: stderr/stdout may or may not interest you:

linux:~/Minefield # ./firefox/firefox -P testing -chat
nsStringStats
 => mAllocCount:             41
 => mReallocCount:           19
 => mFreeCount:              22  --  LEAKED 19 !!!
 => mShareCount:             77
 => mAdoptCount:              1
 => mAdoptFreeCount:          1
linux:~/Minefield #
Comment 40 Tony Mechelynck [:tonymec] 2012-02-21 09:04:30 PST
oops, forgot -no-remote. Trying again.
Comment 41 Tony Mechelynck [:tonymec] 2012-02-21 09:29:07 PST
Still no crash (but a much longer stdout/stderr output).
Comment 42 David Mandelin [:dmandelin] 2012-02-22 17:46:44 PST
(In reply to Tony Mechelynck [:tonymec] from comment #38)
> P.S. My STR are very simple: "1. Start ChatZilla". This crashes every time
> for me. I've tried to attach as much info as I could think of about my cZ
> config in the hope of making the problem reproducible.

I didn't try patching in the config stuff, but I did try starting ChatZilla, and that by itself didn't crash for me.

(In reply to Brian Hackett (:bhackett) from comment #36)
> Can you try these builds (when they appear):
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bhackett@mozilla.
> com-9b408f8a057b
> 
> This just backs out the patch.  I don't see anything wrong with the patch so
> without some solid STR a backout is really the only option.

So what's next? It looks like a pretty serious bug. What about creating some builds with extra diagnostics?
Comment 43 Tony Mechelynck [:tonymec] 2012-02-22 18:42:13 PST
(In reply to David Mandelin from comment #42)
> (In reply to Tony Mechelynck [:tonymec] from comment #38)
> > P.S. My STR are very simple: "1. Start ChatZilla". This crashes every time
> > for me. I've tried to attach as much info as I could think of about my cZ
> > config in the hope of making the problem reproducible.
> 
> I didn't try patching in the config stuff, but I did try starting ChatZilla,
> and that by itself didn't crash for me.

In the crashing builds, I see the crash after ChatZilla quickly opens quite a number of tabs as part of my startup routine. Maybe I had the bad luck to configure ChatZilla with enough “complexity” to crash in post-6-January trunk builds.
Comment 44 Tony Mechelynck [:tonymec] 2012-02-27 17:02:47 PST
(In reply to David Mandelin from comment #42)
[...]
> (In reply to Brian Hackett (:bhackett) from comment #36)
[...]
> So what's next? It looks like a pretty serious bug. What about creating some
> builds with extra diagnostics?

(Ping)
I'm willing to run any tests which I will be told but I don't have the required abilities to devise meaningful tests. I also don't have the resources to compile locally, if a custom build is required, please produce a linux64 try-build.
Comment 45 Brian Hackett (:bhackett) 2012-02-28 10:17:41 PST
Have you been using debug or release builds?  If the latter, can you try a debug build and see if anything asserts?
Comment 46 Tony Mechelynck [:tonymec] 2012-02-28 21:00:33 PST
(In reply to Brian Hackett (:bhackett) from comment #45)
> Have you been using debug or release builds?  If the latter, can you try a
> debug build and see if anything asserts?

I've been using "ordinary" nightlies, and the try-builds which I was asked to test.
Let's try http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-02-28-mozilla-central-debug/firefox-13.0a1.en-US.debug-linux-i686.tar.bz2
Comment 47 Tony Mechelynck [:tonymec] 2012-02-28 21:02:22 PST
P.S. (No linux64 debug build available AFAICT)
Comment 48 Tony Mechelynck [:tonymec] 2012-02-28 21:29:48 PST
Strangely enough,
- no crash in today's Firefox (Nightly) linux-i686 debug nightly (L32 on L64)
- no crash in today's Firefox (Nightly) linux-x86_64 (nondebug) nightly
- no crash in today's SeaMonkey (trunk) linux-x86_64 nightly.

I had been chatting with Fx 3.6 because of this crash. Let's use trunk nightlies again and see if the crash reappears.
Comment 49 Tony Mechelynck [:tonymec] 2012-02-28 21:52:07 PST
Created attachment 601520 [details]
log in Feb-29 SeaMonkey, starting at cZ startup

P.S. I notice a lot of ChatZilla-related messages in the stdout/stderr log for:
Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120228 Firefox/13.0a1 SeaMonkey/2.10a1 ID:20120228003031

Don't know if relevant (about why it doesn't crash), I'm attaching it just in case.
Comment 50 Alex Keybl [:akeybl] 2012-02-29 13:39:52 PST
Even if this bug is resolved/WFM on nightlies, we presumably need to continue the investigation given the affect on FF12.
Comment 51 Tony Mechelynck [:tonymec] 2012-03-04 09:17:30 PST
Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120304 Firefox/13.0a1 SeaMonkey/2.10a1 ID:20120304003026

I still don't see this bug in SeaMonkey trunk despite starting the next nightly (with ChatZilla) every day.

We know with which changeset this problem appeared (see comment #34), we don't know exactly when it disappeared.
Comment 52 David Mandelin [:dmandelin] 2012-03-06 14:16:21 PST
Well, at this point I don't think there's too much more to do. The only thing would be to determine what changeset fixed the problem and see if we want to backport it to 12. We'd need Tony's help. I don't know if it's worth the effort, though.
Comment 53 Tony Mechelynck [:tonymec] 2012-03-06 19:05:43 PST
(In reply to David Mandelin from comment #52)
> Well, at this point I don't think there's too much more to do. The only
> thing would be to determine what changeset fixed the problem and see if we
> want to backport it to 12. We'd need Tony's help. I don't know if it's worth
> the effort, though.

I'm still ready to help but I need directions and/or L64 try-builds.

IIRC this issue (unless fixed in time, of course) will be mentioned under "Known Issues" on the SeaMonkey 2.9 release notes page (Callek, correct me if I'm wrong).
Comment 54 Misak Khachatryan 2012-03-07 03:30:00 PST
I've started to hit this from last merge in mozilla-central. Lunix 64bit SeaMonkey. Volunteering to help too.
Comment 55 Jens Hatlak (:InvisibleSmiley) 2012-03-07 14:06:37 PST
(In reply to Tony Mechelynck [:tonymec] from comment #53)
> IIRC this issue (unless fixed in time, of course) will be mentioned under
> "Known Issues" on the SeaMonkey 2.9 release notes page (Callek, correct me
> if I'm wrong).

Known Issues is purely constructed manually. Best is to always CC me and add the relnote keyword in such cases (where SM is affected).
Comment 56 David Mandelin [:dmandelin] 2012-03-07 16:27:20 PST
Thanks for volunteering, Tony and Misak. The first thing would be to find the day on which it started working again using nightlies. I don't think we know that yet. Can you bisect?

After that, if we're lucky, we'll just be able to spot the fix in that regression range, otherwise I'll prepare try builds to narrow it down just like before.
Comment 57 David Mandelin [:dmandelin] 2012-03-07 19:03:34 PST
Marking assignee since I'm the one working on this from our end.
Comment 58 Tony Mechelynck [:tonymec] 2012-03-08 12:14:01 PST
(In reply to Jens Hatlak (:InvisibleSmiley) from comment #55)
> (In reply to Tony Mechelynck [:tonymec] from comment #53)
> > IIRC this issue (unless fixed in time, of course) will be mentioned under
> > "Known Issues" on the SeaMonkey 2.9 release notes page (Callek, correct me
> > if I'm wrong).
> 
> Known Issues is purely constructed manually. Best is to always CC me and add
> the relnote keyword in such cases (where SM is affected).

Ah, OK. I thought it had been mentioned at the Sm Status Meeting but maybe no firm decision taken.
Comment 59 Tony Mechelynck [:tonymec] 2012-03-08 12:26:08 PST
(In reply to David Mandelin from comment #56)
> Thanks for volunteering, Tony and Misak. The first thing would be to find
> the day on which it started working again using nightlies. I don't think we
> know that yet. Can you bisect?
> 
> After that, if we're lucky, we'll just be able to spot the fix in that
> regression range, otherwise I'll prepare try builds to narrow it down just
> like before.

Using Fx linux x86_64 nightlies
2012-01-08 03:10:24 is Bad,  see comment #9
2012-02-28          is Good, see comment #48
Comment 60 Tony Mechelynck [:tonymec] 2012-03-08 12:38:31 PST
2012-02-06 03:11:48 rev 814d0b2dbaba is Bad: bp-177015dc-0ce2-4757-8148-2a5b12120308
Comment 61 Tony Mechelynck [:tonymec] 2012-03-08 12:47:18 PST
2012-02-17 03:12:27 rev 2271cb92cc05 is Bad: bp-518e30c9-8c46-4d29-b620-bbb0c2120308
Comment 62 Tony Mechelynck [:tonymec] 2012-03-08 12:56:14 PST
2012-02-22 03:12:23 rev 373c710112e6 is Good
Comment 63 Tony Mechelynck [:tonymec] 2012-03-08 13:03:41 PST
2012-02-20 07:49:32 rev 0a7410527788 is Good
Comment 64 Tony Mechelynck [:tonymec] 2012-03-08 13:11:35 PST
2012-02-18 03:11:56 rev 550779e6bab4 is Good (first known good)
2012-02-17 03:12:27 rev 2271cb92cc05 is Bad (last known bad), see comment #61
Comment 65 David Mandelin [:dmandelin] 2012-03-09 19:07:29 PST
(In reply to Tony Mechelynck [:tonymec] from comment #64)
> 2012-02-18 03:11:56 rev 550779e6bab4 is Good (first known good)
> 2012-02-17 03:12:27 rev 2271cb92cc05 is Bad (last known bad), see comment #61

Thanks! Unless I've made a mistake somewhere:

- The regressing changeset was faf5f8842fec, which is present on mozilla-aurora (now 12) and mozilla-central (now 13).
- The fixing changeset landed on Feb 17, at which time mozilla-central was 13.
- The exact fixing changeset might be 75eb6956410b, which I also confirmed is present on mozilla-central but not mozilla-aurora

=> 11 never had the bug
=> 12 has the bug
=> 13 is fixed

I was going to send a try build to home in on the fix so we can try to get it for 12, but my attempts to pull a new aurora are failing (maybe it's down for maintenance or something), so I'll do that next week.
Comment 66 David Mandelin [:dmandelin] 2012-03-16 12:45:13 PDT
OK, the beta is now version 12, and I verified that the code for 75eb6956410b is in the build. So I believe this is now fixed on all versions. Tony, would you mind doing the honors and verifying in the new beta?
Comment 67 Tony Mechelynck [:tonymec] 2012-03-16 15:33:13 PDT
(In reply to David Mandelin from comment #66)
> OK, the beta is now version 12, and I verified that the code for
> 75eb6956410b is in the build. So I believe this is now fixed on all
> versions. Tony, would you mind doing the honors and verifying in the new
> beta?

20120314195616
http://hg.mozilla.org/releases/mozilla-beta/rev/4027017bbaba

This is the current Firefox 12.0 beta candidate (12.0b1 build2).

No crash.
Comment 68 David Mandelin [:dmandelin] 2012-03-16 17:56:32 PDT
(In reply to Tony Mechelynck [:tonymec] from comment #67)
> (In reply to David Mandelin from comment #66)
> > OK, the beta is now version 12, and I verified that the code for
> > 75eb6956410b is in the build. So I believe this is now fixed on all
> > versions. Tony, would you mind doing the honors and verifying in the new
> > beta?
> 
> 20120314195616
> http://hg.mozilla.org/releases/mozilla-beta/rev/4027017bbaba
> 
> This is the current Firefox 12.0 beta candidate (12.0b1 build2).
> 
> No crash.

Thanks for all your help!
Comment 69 Tony Mechelynck [:tonymec] 2012-03-17 05:35:16 PDT
(In reply to David Mandelin from comment #68)
[...]
> Thanks for all your help!

Without you I couldn't have done anything. This bug (crashing me at every startup of ChatZilla) hit me hard enough that I wanted it to go away and stay away.

A last thought: Does it deserve tests so that it doesn't reoccur? Or are there already enough tests for bug 727330 (which gives me "access denied")?
Comment 70 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-05-30 10:22:51 PDT
Tony, could you help us by testing Firefox 13.0b6 to see if the crash reproduces? Thanks.
Comment 71 Tony Mechelynck [:tonymec] 2012-05-31 22:23:31 PDT
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #70)
> Tony, could you help us by testing Firefox 13.0b6 to see if the crash
> reproduces? Thanks.

Do you have a download URL for Linux64? The versions I routinely download by FTP are the latest 15.0a1 nightly and the latest Namoroka (currently 3.6.29pre).
Comment 72 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-05-31 22:33:50 PDT
Sure, all 13.0b6 can be found here:
ftp://ftp.mozilla.org/pub/firefox/releases/13.0b6/
Comment 73 Tony Mechelynck [:tonymec] 2012-05-31 23:58:08 PDT
Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0
Built from http://hg.mozilla.org/releases/mozilla-beta/rev/5dd4dde1d4eb
ChatZilla 0.9.88.2 from current AMO

No crash, not even after copying <profile>/chatzilla/ directory and contents, and extensions/irc/* preferences, as in comment #9 above.

==> verified-fx13
Comment 74 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-06-01 09:07:26 PDT
Thanks Tony.

Note You need to log in before you can comment on or make changes to this bug.