466531 - Crash/hang [@ mult][@ Balloc] when loading pages on PPC

Status: VERIFIED FIXED

(Core :: JavaScript Engine, defect)

Description

[Chris Lawson (gone)]

Attachment: crash log from 23 November nightly

STR:

1) With a clean profile, visit http://blogs.herald.com/ with any build after 14 November (which works fine; 15 November and subsequent builds die horribly).
2) Wait a minute.

ER: Site loads properly.

AR: Kaboom.

The attached crash log is the only "clean" one I was able to get (which comes from a 23 November build). Everything after that has had Talkback all up in its mix, despite a restart and the disabling of Talkback in the active Camino package.

Comment 1

[David Ryeburn]

Attachment: 2008-11-15 Nightly version crash log

Comment 2

[David Ryeburn]

Using Troubleshoot Camino I went to http://www.nytimes.com/ which produces this type of crash for versions November 15 and later. November 14 is OK even using my usual profile.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

The regression range (module Camino) has only two checkins, the NSPR upgrade (bug 463073) and bug 444103 (which isn't built in Camino—aside from perhaps that idl—since we're still using Talkback).  Even expanding that to all checkins across the tree, there's nothing else besides bug 463073 that seems likely to cause this.

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-11-14+00%3A00&maxdate=2008-11-15+00%3A00&cvsroot=%2Fcvsroot

When I look at Talkback for Camino on 1.9.0, we suddenly have crashes in "mult" and "Balloc" beginning on 15 Nov.  Similarly, crash-stats reports that Firefox has suddenly seen a large number of crashes in "mult" beginning on 15 Nov, only on Mac OS X and only on PPC: 

http://crash-stats.mozilla.com/report/list?product=Firefox&branch=1.9&version=Firefox%3A3.0.5pre&platform=mac&query_search=signature&query_type=contains&query=mult&date=&range_value=2&range_unit=weeks&do_query=1&signature=mult

Note that the Mac OS X crash logs attached to this bug mis-identify the crashing thread; we've seen certain types of "nasty" crashes confuse the Mac OS X Crash Reporter before, and both Talkback and Breakpad seem to agree the crashing thread is the one with "mult" (and also "Balloc" in the Camino case).

Requesting blocking since this is a recent crash/hang regression.

wtc, I assume that the underlying cause of this crash (the functions mentioned in frame 0 are actually quoted as coming from libmozjs) is something in that NSPR upgrade and this bug needs to live there; can you confirm and move?

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Sample Talkback reports from Camino

Here are some sample crashing threads from Camino's Talkback, just for easy access (crash-stats is a model of stability compared to Talkback these days).  The crash reports for Firefox that I looked at on crash-stats are not exactly the same, but the first few frames are very similar.

Comment 6

[Samuel Sidler (old account; do not CC)]

This should probably block. It's currently the topcrash on the 1.9.0 branch and it clearly regressed in the builds starting on the 15th.

Since the stacks seem to indicate that this is JS crash, I'm moving it there, even though there's no evidence that a JS checkin could've caused this. Maybe the NSPR update caused a JS bug to become visible?

Comment 7

[Samuel Sidler (old account; do not CC)]

This crash doesn't exist on 1.9.1, afaict. But NSPR 4.7.2 has landed there (4.7.3 hasn't though, but that should've been Sun changes only (bug 461502).

The only real Mac-affecting bugs in NSPR 4.7.2 were:
  * bug 417044
  * bug 454878
  * bug 455829

Bug 455829 scares me a little because there was talk of #ifdef-ing LINUX which doesn't look like it was done, but I'm still not sure that would trigger this code path...

Comment 8

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

FWIW, I do also see a few Balloc crashes with similar stacks in Firefox (I can't convince crash-stats to complete a query for two weeks worth of data, though): http://crash-stats.mozilla.com/report/list?product=Firefox&branch=1.9&platform=mac&query_search=signature&query_type=contains&query=Balloc&date=&range_value=1&range_unit=weeks&do_query=1&signature=Balloc

Also, other trigger URLs reported by Camino users in the forum and by Camino users in Talkback report comments include:

http://www.nytimes.com - Viewed the front page
http://www.google.com - clicking on gmail
http://facebook.com - loading the canvas page (home)
http://en.wikipedeia.org - opening an article
http://www.cnn.com - open 3 tabs; cnn abc & google news pages
http://cnn.com - tying to open cnn.com
http://www.excite.com
http://www.weatherbug.com - Checking our local weather
http://weather.weatherbug.com - Linking to another page of local weather
http://forums.mozillazine.org/viewforum.php?f=12 - requesting that url

Firefox 3.0.5pre reports on crash-stats (where URLs are not reported) have a few comments; those seem to indicate crash-on-launch/crash-on-relaunch scenarios.

Comment 9

[Chris Lawson (gone)]

Notably, the default home page in a fresh profile for Camino ( http://caminobrowser.org/welcome/ ) does *not* trigger this bug, if someone is thinking about trying to develop a testcase.

Comment 10

[Wan-Teh Chang]

(In reply to comment #7)
> Bug 455829 scares me a little because there was talk of #ifdef-ing LINUX which
> doesn't look like it was done,

Bug 455829 doesn't affect Mac OS X.

Comment 11

[Samuel Sidler (old account; do not CC)]

Attachment: Sample 1

I can't get GranParadiso to crash on 10.4.11 PPC. Instead, it loads, but hangs before the window loads. The titlebar appears, but by then the app is hung. I tried changing my start page to about:blank (using 3.0.4) to no avail. Here's a sample.

(As an aside, when changing my start page, 3.0.4 crashed. But it crashed in a place we're already planning on fixing for 3.0.6.)

Comment 12

[Samuel Sidler (old account; do not CC)]

Replacing libmozjs.dylib in GranParadiso.app/Contents/MacOS with the one from Firefox 3.0.4 removes the hang for me. Adding back the one from 3.0.5pre brings the hang back... (Using 10.4.11 PPC as in comment 11.)

Comment 13

[Wan-Teh Chang]

Attachment: NSPR public header changes between NSPR_4_7_1_RTM and NSPR_4_7_3_RTM

The only NSPR changes that may affect libmozjs.dylib are the changes to
NSPR's public header files.  I reviewed them again, and didn't see anything
that could break libmozjs.dylib.  The changes are:
- removal of XP_OS2_VACPP support
- addition of SYMBIAN support
- addition of 64-bit Mac OS X (x86_64) support

Comment 14

[Samuel Sidler (old account; do not CC)]

Can we try backing out the Mac OS X 64-bit change and produce a try-server build?

Comment 15

[David Ryeburn]

After reading Samuel Sidler's Comment #12 I transplanted the libmozjs.dylib
file from Firefox 3.0.4 to Camino 2.0b1pre 20081115 (the earliest version which
misbehaves for me at the NY Times main page). This appeared to work fine, even
when using my regular profile instead of starting with Troubleshoot Camino.
Next I tried the same thing with Camino 2.0b1pre 20081124 (after first
verifying that without the transplant it misbehaved the same way, even using
Troubleshoot Camino). It appeared to work just as well as the modified 20081115
version did, until I opened a video from the main page. When that video
finished, it switched automatically to another one. I couldn't pause it, nor
close the main page, nor quit Camino without force quitting. At this point I
tried the same experiment with the 20081115 version (with transplant). It too
balked after playing the first video when I tried to pause the second one.
Perhaps this is a different problem from the apparent Javascript problem. The
unmodified 20081114 version, which is the last one which appears to behave
normally, plays the video and doesn't misbehave afterwards when I pause the
next one. One interesting thing which I noticed is that the problem with
modified 20081115 and later doesn't arise until I try to pause the video.
Presumably if I were willing to let the NY Times feed me one video after
another without interference, it would work OK.

Comment 16

[Daniel Veditz [:dveditz]]

The other interesting thing to try would be to transplant a 3.0.4 NSPR into a broken Camino/Firefox3.0.5 since the regression range fingers NSPR. Sam reported that that didn't help Firefox (which is why he tried the js lib) but it'd be good to get a Camino report too. 

There were very limited JS checkins, and none during the supposed 11/14-11/15 regression period. Are we sure about that time range?
http://bonsai.mozilla.org/cvsquery.cgi?module=all&dir=mozilla%2Fjs%2Fsrc&date=explicit&mindate=2008-11-01&maxdate=2008-11-24&cvsroot=%2Fcvsroot

Comment 17

[Chris Lawson (gone)]

(In reply to comment #16)
> There were very limited JS checkins, and none during the supposed 11/14-11/15
> regression period. Are we sure about that time range?

I'm 100% confident that's the right regression range. Comment 6 seems like the only plausible theory so far :-\

I'll try moving NSPR from a Camino nightly prior to the problem into an affected nightly and report back.

Comment 18

[David Ryeburn]

Daniel Veditz (Comment #16) suggested transplanting a 3.0.4 NSPR into Camino. I'd be glad to do that if you tell me which pieces of Firefox 3.0.4 to transplant; I'm a (retired) mathematician, not a computing person, and don't know what NSPR is.

I'm absolutely sure that Camino 2.0b1pre 20081114 without any modification works fine (so far) including tolerating pausing videos at the NY Times site, that 20081115 as it comes from the factory fails while loading the NY Times main page, that it appears to be fixed upon transplanting libmozjs.dylib from Firefox 3.0.4, but that when I try to pause a video at the NY Times site it then misbehaves. I didn't try every version but looked between Camino 2.0a1 2008101818 (behaves nicely) and Camino 2.0b1pre 2008117 (earliest I then knew misbehaved loading the NY Times main page), bisecting the interval successively until I narrowed it down to November 14 good and November 15 bad. The transplant is unnecessary for the November 14 version. I tried it with the partial success indicated above first on the November 15 version and then on the November 24 version, but not on any versions in between. All in all I've only looked at maybe 20% of the versions from October 18 through November 24, while trying to pin down where the break was. So maybe some of the other 80% I didn't check don't behave the way their positions in the sequence would indicate.

Comment 19

[Wan-Teh Chang]

NSPR consists of three files: libnspr4.dylib, libplc4.dylib,
and libplds4.dylib.  Thanks.

Comment 20

[Chris Lawson (gone)]

Here are the results of my testing. I don't have the 15 November nightly around any more, but I expect it wouldn't make any difference, as I was getting the crash every time I tried to reproduce it earlier with that build.


With the 14 November NSPR and the 14 November libmozjs (a "normal" 14 November Camino nightly), no crash.

With the 23 November NSPR and the 23 November libmozjs (a "normal" 23 November Camino nightly), I get this crash.

With the 14 November NSPR and the 23 November libmozjs in what is otherwise a "normal" 23 November Camino nightly, I get this crash.

With the 23 November NSPR and the 14 November libmozjs in what is otherwise a "normal" 23 November Camino nightly, there is no crash.

It sure sounds to me like that fingers libmozjs, but as Daniel pointed out, none of those checkins fall within the regression range of (approximately) 0000 on 14 November through 1000 on 15 November.

Comment 21

[Chris Lawson (gone)]

The command-line diff tool just says "binary files differ" when diffing the libmozjs.dylib from 14 November against 15 November, but TextWrangler shows a couple places where path information is different:

Comment 22

[Chris Lawson (gone)]

(In reply to comment #21)
> The command-line diff tool just says "binary files differ" when diffing the
> libmozjs.dylib from 14 November against 15 November, but TextWrangler shows a
> couple places where path information is different:

...and apparently Bugzilla didn't like the characters from the paste, so it cut off the rest of my comment. The differences look something like this:

Comment 23

[Chris Lawson (gone)]

Argh. Did it again.

http://pastebin.mozilla.org/577500

shows the (cleaned-up) path information. It's a bunch of NSPR-related paths, interestingly enough. Not sure if that's coincidence or not; the paths themselves don't seem to have changed from the 14th to the 15th, but the stuff between paths (at least, between path information that's human-readable) did.

Comment 24

[David Ryeburn]

Chris Lawson in Comment #20 indicated that the November 23 NSPR and November 14 libmozjs.dylib combination in the November 23 Camino version didn't produce a crash, whereas I had indicated the the November 24 NSPR and November 14 libmozjs.dylib combination in the November 24 Camino version *did* eventually produce a crash, after I paused a NY Times video. (Until looking at the video, that combination seemed to be working OK.) I went back, replaced the Firefox 3.0.4 version of libmozjs.dylib with the November 14 Camino version, and also replaced the NSPR files (thanks, Wan-Teh Chang, for letting me know which they are) with the ones from the November 14 Camino version. With both transplants installed, I can now pause NY Times videos without problems. I'll keep using this for a while, and also modifying later versions in the same way as they appear.

Chris, did you check the NY Times video for pausability with just the libmozjs.dylib replaced?

Comment 25

[Chris Lawson (gone)]

(In reply to comment #24)

> Chris, did you check the NY Times video for pausability with just the
> libmozjs.dylib replaced?

No.

Comment 27

[Oscar van Vliet]

Followup from bug 466673: I found that the hang occurs immediately on loading http://www.nature.com/nrn/journal/vaop/ncurrent/full/nrn2473.html

Comment 28

[Oscar van Vliet]

So what is the latest 'safe' nightly? This bug is annoying the sh*t out of me and I want to go back to an earlier version that doesn't have it. I assume it is somewhere in the http://ftp.mozilla.org/pub/mozilla.org/camino/nightly/ hierarchy.

Comment 29

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #28)
> So what is the latest 'safe' nightly? This bug is annoying the sh*t out of me
> and I want to go back to an earlier version that doesn't have it. I assume it
> is somewhere in the http://ftp.mozilla.org/pub/mozilla.org/camino/nightly/
> hierarchy.

Oscar, the Nov 14 nightly should be "safe" (2008-11-14-00-Cm2-M1.9)

Comment 30

[Chris Lawson (gone)]

(In reply to comment #28)
> So what is the latest 'safe' nightly?

Without any workarounds, 14 November should be (see comment 0).

Workarounds in comment 12 and comment 20 (see also comment 19) should allow you to use a more recent nightly after some modification.

cl

Comment 31

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

FWIW, I also ran TextWrangler's compare files function on the libmozjs.dylib, and what I noticed is that

1) the 2 @executable_path lines differ "regularly" between builds (that is, the little bits of binary between the human-readable file paths on those lines in the 13th's build differ from the 14th's build, which also differ from the 15th's build); I assume these are "expected" differences.

2) however, there are no other differences in the binary mess of the rest of dylib between the 13th and the 14th

3) but there are 6 "hunks" or sections of the binary mess that differ between the 14th and 15th.

That certainly indicates to me that some change elsewhere in the tree caused "unexpected" changes in libmozjs.

Comment 32

[Oscar van Vliet]

I can confirm that replacing libmozjs.dylib with the one from http://ftp.mozilla.org/pub/mozilla.org/camino/nightly/2008-11-14-00-2.0-M1.9/ fixes the initial problem for me. Thanks for the fast advice! If I run into video problems I will also grab libnspr4.dylib, libplc4.dylib,and libplds4.dylib and see if that helps.

I am assuming Mozilla will not release Firefox 3.0.5 before this is fixed.

I was also going to suggest a note about this on the weblog, but Smokey already took care of that :).

best wishes,
 Oscar

Comment 33

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

wtc, could the patch in bug 370766 for prlink.c that hasn't landed influence this bug in any way?  That bug really looks like it's only dealing with x86_64 changes, anyway, and not PPC-anything, but at the moment we seem to be grasping at straws all over….

Comment 34

[Wan-Teh Chang]

You can change NSPR_CO_TAG back to NSPR_4_7_1_RTM in mozilla/client.mk
(just back out attachment 348227 [details] [diff] [review]).  If that fixes the crash, then we can
try to narrow down which NSPR change is responsible.  I have reviewed
all the NSPR changes and none of them seem to be able to cause this
crash.

Comment 35

[Daniel Veditz [:dveditz]]

CC'ing ted just in case it is in fact his crashreporter change. That would at least explain the mac-only aspect, but it doesn't look like xpcom/base/nsObjCExceptions.h gets included anywhere interesting wrt js/src/

Comment 36

[Stuart Morgan]

Daniel, why did you remove my dependency to the bug we are using for Camino must-fix trackers?

Comment 37

[Oscar van Vliet]

FYI, I tried Vimeo and Youtube in a Nov. 25 Camino nightly with a transplanted Nov. 14 libmozjs.dylib and had no problems. So far, so good.

Comment 38

[Stuart Morgan]

Oscar and David, thanks for the help finding and verifying the regression range. Now that we have it isolated though, we'd like to focus the bug on attempts to find and fix the root problem and keep discussion of temporary workarounds to a minimum (the forums would be a good place for any other discussion of that type). Thanks!

Comment 39

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

OK, I finally have a build with NSPR rolled back to NSPR_4_7_1_RTM.

Can those of you with PPC Macs please download this build and report back as to whether the crash/hang is gone or remains?

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081125-NSPR471.dmg

I'll be back again (much more quickly this time, now that my PPC build is unhorked) with a build that is back to NSPR 4.7.3 again but which has backed out Ted's patch and bustage fixes for bug 444103 instead.

Comment 40

[Chris Lawson (gone)]

(In reply to comment #39)
> OK, I finally have a build with NSPR rolled back to NSPR_4_7_1_RTM.
> 
> Can those of you with PPC Macs please download this build and report back as to
> whether the crash/hang is gone or remains?

Still crashes with the same stack as before.

Comment 41

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I'm on my way out the door, but in about 15 minutes when the upload is done, you can get the no-bug-444103 build from:

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081125-noObjClogging.dmg

Comment 42

[Chris Lawson (gone)]

(In reply to comment #41)
> I'm on my way out the door, but in about 15 minutes when the upload is done,
> you can get the no-bug-444103 build from:

Which also crashed with the same stack as before :-\

Comment 43

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Finally, here's a build that has reverted back to NSPR 4.7.1 again *and* has backed out Ted's patch and bustage fixes for bug 444103:

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081125-NSPR471-noObjClogging.dmg

If it still crashes, I give up :P

I also checked the Firefox build hours (crash-stats reports no crashes in "mult" or "Balloc" on 14 Nov, and many beginning on 15 Nov), and they're built at 0400, after the Camino builds.  I also expanded the checkin range by 4 hours on each side of the Camino builds (to cover 20:00 13 Nov-04:00 15 Nov) and didn't net any new checkins.

Comment 44

[Chris Lawson (gone)]

(In reply to comment #43)
> Finally, here's a build that has reverted back to NSPR 4.7.1 again *and* has
> backed out Ted's patch and bustage fixes for bug 444103:
> 
> http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081125-NSPR471-noObjClogging.dmg
> 
> If it still crashes, I give up :P

Still crashes, but with a new and different stack that might be indicative of even worse things happening:

Process:         Camino [44553]
Path:            /Volumes/Camino/Camino.app/Contents/MacOS/Camino
Identifier:      org.mozilla.camino
Version:         2.0b1pre (2008.11.25)
Code Type:       PPC (Native)
Parent Process:  launchd [155]

Date/Time:       2008-11-25 23:40:35.421 -0500
OS Version:      Mac OS X 10.5.5 (9F33)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000007cd8cf4d
Crashed Thread:  Unknown

Error Formulating Crash Report:
*** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value (key: VMUSignaturePath)
0x937f4dec
0x915334ec
0x937f4cfc
0x937f4d34
0x908cb968
0x00071c00
0x000804d8
0x00080614
0x00003004
0x00009ad4
0x0000b84c
0x0000b63c
0x911139d0
0x0000ad58
0x910e565c

Backtrace not available

Unknown thread crashed with PPC Thread State 32:
  srr0: 0x0116cec4  srr1: 0x0000f030   dar: 0x7cd8cf4d dsisr: 0x40000000
    r0: 0x00000000    r1: 0xf0102ca0    r2: 0x074702f0    r3: 0xd7193d59
    r4: 0x00000000    r5: 0x00000000    r6: 0x492cd342    r7: 0x000dd182
    r8: 0x492cd342    r9: 0x43300000   r10: 0x43300000   r11: 0x00feb217
   r12: 0xedec2290   r13: 0x00000000   r14: 0x00000000   r15: 0x00000000
   r16: 0x00000000   r17: 0x00000000   r18: 0x00000000   r19: 0x00000000
   r20: 0x00000000   r21: 0x00000000   r22: 0x00000000   r23: 0x00000000
   r24: 0x00000000   r25: 0x01198494   r26: 0x80000000   r27: 0x02761704
   r28: 0x027617a0   r29: 0x027616e0   r30: 0x7cd8cf21   r31: 0x01168494
    cr: 0x24044224   xer: 0x20000000    lr: 0x0116cea4   ctr: 0x910b5ff8
vrsave: 0x00000000

Binary images description not available

Comment 45

[Wan-Teh Chang]

Smokey: just to make sure you revert to NSPR_4_7_1_RTM correctly:
you need to do "make -f client.mk checkout" again after changing
mozilla/client.mk.  Alternatively, you can do this:

  cvs -q co -r NSPR_4_7_1_RTM mozilla/nsprpub

Use the command

  ident libnspr4.dylib | grep NSPR

to verify it's "NSPR 4.7.1".

Comment 46

[Chris Lawson (gone)]

All four of the archived hourly builds from 14 November on

http://hourly-archive.localgho.st/mac.html

work just fine. The 15 November GranParadiso nightly crashes.

Comment 47

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #45)
> Smokey: just to make sure you revert to NSPR_4_7_1_RTM correctly:
> you need to do "make -f client.mk checkout" again after changing
> mozilla/client.mk.

Yes, I did that (actually a full "make -f client.mk" after the tag backout), and I watched the cvs messages confirm it was checking out NSPR_4_7_1_RTM each time.

On the other hand, "ident libnspr4.dylib | grep NSPR" on any of builds I made tonight gives me 
     $Header: NSPR 4.7.3  2008-11-25 17:11:43 $
     $Header: NSPR 4.7.1  2008-11-25 18:45:51 $
(I didn't clobber between builds because I'm unfortunately not an Xserve and can't do 45min clobbers) :(

However, to rule out any errors caused by my builds and to double-check the results, Chris also used the 2008-11-15-04 Firefox 3.0.5pre nightly to verify the crash (it crashes on startup with the "expected" stack in the crashing thread; see bp-21474c99-0872-41a2-9e87-6af4b2081125 ).

Chris then used the 2008-11-14 Firefox 3.0.5pre hourly builds from http://hourly-archive.localgho.st/mac.html to step through the day's checkins (the first build covers Ted's patch and bustage fixes for bug 444103; the second build is the NSPR push, the third build is a bustage fix for the crashreporter test API, and the fourth build contains dcamp's safebrowsing checkin for bug 461891).  None of the four builds crashed for him.  

According to bonsai, the next checkin for Firefox 3.0.5pre after the 2008-11-14 1513 hourly was not until 2227 on the 15th, well after the build time of the 2008-11-15 0400 nightly that crashes on startup (there was a mozilla/security/nss checkin at 1006 on the 15th, but that's NSS trunk, not client.mk-controlled cvs).

Oh, *but* "ident libnspr4.dylib | grep NSPR" reports 
     $Header: NSPR 4.7.1  2008-11-14 04:06:35 $
     $Header: NSPR 4.7.1  2008-11-14 04:23:03 $
on *all four* of those Firefox 3.0.5pre hourly builds, which means that, as non-clobber builds, they don't seem to have "really" picked the NSPR change?

I guess I'm going to try to spin a clobber build or a fresh tree overnight with just the NSPR backout.  Sorry for all the noise caused by my sloppy build-wrangling :-(

Comment 48

[Daniel Veditz [:dveditz]]

(In reply to comment #36)
> Daniel, why did you remove my dependency to the bug we are using for Camino
> must-fix trackers?

I don't know how I did that -- sorry! I'd blame a silent-midair (sometimes bugzilla doesn't mention dependency bug conflicts) but at 6hrs later that seems unlikely!

Comment 49

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #47)
> I guess I'm going to try to spin a clobber build or a fresh tree overnight with
> just the NSPR backout.  Sorry for all the noise caused by my sloppy
> build-wrangling :-(

Fresh tree+backout of NSPR 4.7.3 tag+verification that NSPR 4.7.1 was pulled+verification that NSPR 4.7.1 was built:

ident libnspr4.dylib | grep NSPR
     $Header: NSPR 4.7.1  2008-11-26 01:41:55 $
     $Header: NSPR 4.7.1  2008-11-26 02:08:28 $

The build finished right as I was getting ready to sleep; it should be online in about 15 minutes at:

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-NSPR471.dmg

(I've also moved the other three older "confused" builds aside so that no one wastes time downloading them.)

Comment 50

[David Ryeburn]

(In reply to comment #49

It works fine for me, running Mac OS 10.4.11 on a slow Digital Audio G4 (466 MHz). It behaves OK at the NY Times main web page, and videos started from there can be paused without hassle. Sorry I didn't check the earlier versions this evening -- we were out at a concert and just got home.

Comment 51

[(not currently active) Ted Mielczarek]

nsObjCExceptions.h only gets included in ObjC files. I don't see any reason this would cause any problems, but you'd really want to ask Josh, as most of the interesting bits there are his.

Comment 52

[Wan-Teh Chang]

Attachment: NSPR public and internal header changes between NSPR_4_7_1_RTM and NSPR_4_7_3_RTM

I found that I made a minor mistake when creating the NSPR_4_7_3_RTM
tag.  The effect is that after NSPR is upgraded from NSPR_4_7_1_RTM
to NSPR_4_7_3_RTM, not all the files that include updated headers
will be recompiled in *non-clobber* builds.  This does not affect
fresh new builds or clobber builds.

This explains the "NSPR 4.7.1" version returned by 'ident' on
libnspr4.dylib from non-clobber builds after the NSPR_4_7_3_RTM
upgrade.

This patch contains all the (public and internal) header changes
between NSPR_4_7_1_RTM and NSPR_4_7_3_RTM.  I reviewed them
carefully, and believe that prinit.h (the NSPR version numbers
and string) is the only problematic one if the .c files that
include it are not recompiled.  So except for the incorrect
version number reported by 'ident', this mistake of the
NSPR_4_7_3_RTM tag should have no ill effects on non-clobber
builds.

Comment 53

[Wan-Teh Chang]

Smokey: I suggest two more experimental builds.  Please
use the tree you pulled in comment 49 so that we can
recreate the NSPR tag upgrade.

1. Change the NSPR tag back to NSPR_4_7_3_RTM.  Pull
NSPR 4.7.3.  Then do a *non-clobber* build.  This
recreates a build in which not all NSPR files are
recompiled.

2. Do a clobber build.  To save time, you only need to
clobber NSPR -- just do "make realclean" in the
nsprpub directory of your build tree before building
the whole tree.

Thanks.  I will borrow an iBook G4 and try to debug
this.

Comment 54

[Chris Lawson (gone)]

(In reply to comment #49)

> Fresh tree+backout of NSPR 4.7.3 tag+verification that NSPR 4.7.1 was
> pulled+verification that NSPR 4.7.1 was built:
> 
> ident libnspr4.dylib | grep NSPR
>      $Header: NSPR 4.7.1  2008-11-26 01:41:55 $
>      $Header: NSPR 4.7.1  2008-11-26 02:08:28 $
> 
> The build finished right as I was getting ready to sleep; it should be online
> in about 15 minutes at:
> 
> http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-NSPR471.dmg

That one's working fine, so we finally (!) know that it does indeed seem to be the NSPR changes that broke this (right?).

Comment 55

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks for the guidance here, Wan-Teh.  I've built and uploaded the two additional experimental builds as requested:

1) The tag-change build is 

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-dep-NSPR473.dmg

and it reports: 

ident libnspr4.dylib | grep NSPR
     $Header: NSPR 4.7.1  2008-11-26 01:41:55 $
     $Header: NSPR 4.7.1  2008-11-26 02:08:28 $

2) The NSPR-clobber ("make realclean" in objdir/ppc/nsprpub and objdir/i386/nsprpub before rebuilding) build is 

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-clb-NSPR473.dmg

and it reports: 

ident libnspr4.dylib | grep NSPR
     $Header: NSPR 4.7.3  2008-11-26 18:01:57 $
     $Header: NSPR 4.7.3  2008-11-26 18:06:05 $


(Just in case we need it, I'm now going to kick off a full clobber build of that tree, since it didn't look like libmozjs.dylib was rebuilt after the nspr-clobber, but libmozjs.dylib was rebuilt after the 4.7.1->4.7.3 upgrade (the dep build), and libmozjs.dylib seems to be the binary that's the source of our troubles, per the early discussion in this bug.)

Comment 56

[Wan-Teh Chang]

Smokey: thanks a lot!  I owe you big time.  I have borrowed
an iBook G4.  I will test the four builds you prepared tonight
(US Pacific Time).  It's a great idea to do a full clobber
build.  Thank you for noticing that libmozjs.dylib wasn't
rebuilt.

I'm not sure if the iBook has the version of Xcode required for
building Camino.  I can build NSPR on it, and will drop my
NSPR builds into your Camino builds for testing, too.  I'm
worried that the bug is in some interaction between libmozjs.dylib
and NSPR (headers or .dylib's).  If that's the case, then I'll
need to learn how to build Camino.

Since NSPR_4_7_1_RTM and NSPR_4_7_3_RTM are two snapshots of
the NSPR trunk, one can identify the problematic checkin by
a binary search (the bisection method).  Hopefully we can
do the binary search by rebuilding just NSPR.

Comment 57

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #56)
> I'm not sure if the iBook has the version of Xcode required for
> building Camino.  

We require Xcode 2.4 or later, but I think any Xcode >= 2.1 will actually work if you disable our GCC_TREAT_WARNINGS_AS_ERRORS build flag (there are apparently warnings produced by older versions of Xcode 2.x that are absent in 2.4).  More info is available http://wiki.caminobrowser.org/Development:Building

That said, this crash is reproducible in Firefox 3.0.5pre as well (Firefox just crashes on startup, based on the crash-stats comments I've seen, and Chris's comment on irc last night when he was doing the research behind comment 46) if you're more familiar with building Firefox.  (I only built Camino because we had Camino users reporting the bug and because I thought I was going to be able to use my pre-existing Universal tree to speed the generation of the builds.)

I will post the full clobber build in a few hours.

Comment 58

[Chris Lawson (gone)]

(In reply to comment #55)
> Thanks for the guidance here, Wan-Teh.  I've built and uploaded the two
> additional experimental builds as requested:
> 
> 1) The tag-change build is 
> 
> http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-dep-NSPR473.dmg

That build works fine.

> 2) The NSPR-clobber ("make realclean" in objdir/ppc/nsprpub and
> objdir/i386/nsprpub before rebuilding) build is 
> 
> http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-clb-NSPR473.dmg

That build *also* works fine.

So, uh, where does that leave us now? I was expecting one or both of these to exhibit the same crash.

Comment 59

[David Ryeburn]

I checked both the tag-change build and the clobber build. I'm using the latter
right now to post this. Both seem to work OK on the NY Times web page and to
allow pausing videos. Which, if either, would you like me to use when I do
other things?

Comment 60

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #57)
> I will post the full clobber build in a few hours.

This was made in the same tree as before with "make -f client.mk clobber build".

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081126-fullclb-NSPR473.dmg

I suspect (hope) that this build will crash for the PPC folks.

I was distracted and wasn't paying attention to watch libmozjs.dylib was rebuilt, but if it wasn't, we have bigger problems with the entire Mozilla build system ;)

Comment 61

[Wan-Teh Chang]

Smokey: I have been working on the iBook G4 for about an hour.  I tested your
last two builds.  As you expected, Camino-Cm2-M1.9-20081126-clb-NSPR473.dmg
works, but Camino-Cm2-M1.9-20081126-fullclb-NSPR473.dmg crashes.  (My
test case is http://www.nytimes.com/.)

This means the mistake in the NSPR_4_7_3_RTM tag that I described in comment 52
is very minor and only causes a non-clobber build to have the old "NSPR 4.7.1" version
as reported by the 'ident' command.

Another conclusion is that it is most likely a change in the NSPR public headers,
or the way libnspr4.dylib is built, that breaks some other part of Firefox/Camino
(presumably libmozjs.dylib).  I will review the diffs again and report back.

Comment 62

[David Ryeburn]

Attachment: Extract from Exited process.crash.log

Comment 63

[David Ryeburn]

(In reply to Comment #60)
> I suspect (hope) that this build will crash for the PPC folks.

It did. The NY Times web page failed to complete loading. After beachballing for about a minute, the application quit. No "quit unexpectedly" message was seen. There's nothing in Camino.crash.log but an extract from Exited process.crash.log is attached.

Comment 64

[Wan-Teh Chang]

Smokey: I need to ask you another favor.

Please attach the jscpucfg.h file in your build tree.
If there are multiple copies of jscpucfg.h, please attach
them all, with their full pathnames.  Since you're building
the Universal binary, I suspect there is a copy for the ppc
build and a copy for the x86 build.

Could you also do a clobber build, save the build log, and
attach the compiler command lines and any compiler warnings
for jsdtoa.c?  (jsdtoa.c should be compiled twice in a Universal
binary build.)

I suspect one of the changes to
mozilla/nsprpub/pr/include/md/_darwin.cfg is the culprit:

@@ -43,22 +43,70 @@
 #endif

 #define PR_AF_INET6 30  /* same as AF_INET6 */

-#if defined(i386)
+#if defined(__i386__) || defined(__x86_64__)
 #undef IS_BIG_ENDIAN
 #define  IS_LITTLE_ENDIAN 1
 #else
 #undef IS_LITTLE_ENDIAN
 #define  IS_BIG_ENDIAN 1
 #endif
...

The change from defined(i386) to defined(__i386__) could
be problematic, although I don't know why it would affect
the ppc build.  I will attach a patch that backs out this
change (it requires a corresponding change to
mozilla/nsprpub/configure).

Comment 65

[Stuart Morgan]

(In reply to comment #64)
> The change from defined(i386) to defined(__i386__) could
> be problematic, although I don't know why it would affect
> the ppc build.

Is it possible that __i386__ is defined to 0 explicitly in PPC builds for some reason? It seems like |defined| is probably the wrong test there...

Comment 66

[Wan-Teh Chang]

Attachment: Patch for NSPR_4_7_3_RTM: change __i386__ back to i386

This experimental patch changes defined(__i386__) back to
defined(i386) in mozilla/nsprpub/pr/include/md/_darwin.cfg.
Smokey, please apply this patch to NSPR_4_7_3_RTM and do a
*full* clobber build.  (Code outside nsprpub also needs to
be recompiled.)

Stuart: Mac OS X system headers (e.g., <math.h>) also test
__i386__ using defined(__i386__).  So it is the common way
to test such architecture macros.

If anyone can point me to a tinderbox that does clobber build
of Mac OS X universal binary, I will be able to get the answers
to some of my questions.  Thanks.

Comment 67

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: jscpucfg.h from the tree used for Smokey's builds

> Please attach the jscpucfg.h file in your build tree.
> If there are multiple copies of jscpucfg.h, please attach
> them all, with their full pathnames.  Since you're building
> the Universal binary, I suspect there is a copy for the ppc
> build and a copy for the x86 build.

I only have one copy of that file in this tree ("trunk-crash"), at /Users/smokey/Camino/dev/trunk-crash/mozilla/js/src/jscpucfg.h.  That is, this file seems to live in the srcdir and is not auto-generated or otherwise copied into the objdirs for the ppc or i386 builds.  It has a modification date of 12:11 AM on 20 Feb 2008.

> Could you also do a clobber build, save the build log, and
> attach the compiler command lines and any compiler warnings
> for jsdtoa.c?  (jsdtoa.c should be compiled twice in a Universal
> binary build.)

I'll kick that off now, but it won't finish before I sleep; I'll post the log if I get a chance tomorrow.  If you're not particularly concerned with the exact results that match these builds I've been making, you should be able to pull that info from the logs of the official tinderboxen.  

The most recent 1.9.0-branch Camino clobber nightly (2008-11-26-00) log is http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1227689160.1227691976.3394.gz&fulltext=1 and the most recent 1.9.0-branch Firefox clobber nightly (2008-11-26-04) is http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1227701100.1227703619.1036.gz&fulltext=1

Comment 68

[Wan-Teh Chang]

D'oh :-)  Please search for the jsautocfg.h files and attach them.

Thanks for the clobber tinderbox URLs.

Comment 69

[Wan-Teh Chang]

After looking at the Firefox clobber nightly build log, I think I found the problem:
jsautocfg.h for the ppc build is incorrectly generated when _darwin.cfg from
NSPR_4_7_3_RTM is used.  I'll be able to verify this when Smokey attaches the
jsautocfg.h for the ppc build.  I bet it says
  #define IS_LITTLE_ENDIAN 1
  #undef  IS_BIG_ENDIAN
which is wrong for ppc, a big-endian processor.

Comment 70

[Wan-Teh Chang]

Here are the commands that compile jscpucfg.c and generate jsautocfg.h for the ppc build.
Note the -DCROSS_COMPILE=1 and -Ui386.  Also note the lack of -arch, which implies
-arch i386 because the build machine is i386.

gcc-4.0 -DMDCPUCFG=\"md/_darwin.cfg\" -DNO_X11 -O3 -DCROSS_COMPILE=1 [...]
-DXP_UNIX=1 -DUNIX_ASYNC_DNS=1 -DJS_THREADSAFE=1 [...]  -Ui386 -DOSTYPE=\"Darwin\" -DOSARCH=Darwin -DEXPORT_JS_API  -DJS_USE_SAFE_ARENA -I../../dist/include/nspr -o jscpucfg /builds/tinderbox/Cm2-M1.9/Darwin_8.10.2_Depend/mozilla/js/src/jscpucfg.c

./jscpucfg > jsautocfg.tmp
mv jsautocfg.tmp jsautocfg.h

In fact, the JS developers knew about the #if defined(i386) check in
_darwin.cfg and depend on it:
http://mxr.mozilla.org/firefox/source/js/src/Makefile.in#415

Their comment that
  [the i386 macro] is also automatically defined by the compiler when
  building for i386.
is wrong.  The compiler doesn't predefine the i386 macro.

Comment 71

[Samuel Sidler (old account; do not CC)]

This blocks 1.9.0.5. We'll either take a patch to NSPR 4.7.3 or revert back to NSPR 4.7.1. I'll confer with Dan later this week.

Comment 72

[Wan-Teh Chang]

It's fine to revert to NSPR 4.7.1 for 1.9.0.5.

The bug is in JS, so the proper fix is to fix jscpucfg.c
and the makefiles used to compile it, rather than patching
NSPR 4.7.3.  We should give the JS team enough time to
implement a proper fix.  I have thought about how to fix
this and will be happy to discuss my ideas with the JS
team.

Does 1.9.1 have the same problem?  If you give me the URL
for a 1.9.1 clobber build tinderbox (for Mac OS X Universal
binary), I can try and find that out.

Comment 73

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: jsautocfg.h from the ppc objdir

This is /Users/smokey/Camino/dev/trunk-crash/CaminoStaticUniTrunk/ppc/js/src/jsautocfg.h (the objdir for the ppc half of the build)

Comment 74

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: jsautocfg.h from the i386 objdir

This is /Users/smokey/Camino/dev/trunk-crash/CaminoStaticUniTrunk/i386/js/src/jsautocfg.h (the objdir for the i386 half of the build)

TextWrangler informs me the two files are identical.

Comment 75

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: compiler commands from (last night's) jsdtoa.c compile

I failed to get the warnings/errors to end up in the logfile (they stayed in the Terminal window and even the i386 one is too far back to still be in the buffer).

(In reply to comment #66)
> Smokey, please apply this patch to NSPR_4_7_3_RTM and do a
> *full* clobber build.  (Code outside nsprpub also needs to
> be recompiled.)

I'll start this now.

Comment 76

[Samuel Sidler (old account; do not CC)]

What kind of a fix does the JS team need to make here?

Comment 77

[Wan-Teh Chang]

Attachment: Proposed patch for JS

Here is the patch I propose for JS.  Because of the difficulty
of testing for endianness of the target processor when we cross
compile, jscpucfg.c can generate code that tests predefined
macros for processor architectures.

Smokey, could you test this patch with NSPR_4_7_3_RTM in a
full clobber build?

There are other possible fixes.

1. When cross compiling for ppc, compile jscpucfg.c with the
-arch ppc flag and without the -DCROSS_COMPILE flag, and run the
resulting jscpucfg ppc binary on Intel, relying on Mac OS X's
Rosetta PowerPC binary translation.  I like this approach the
best but don't have time to figure out how to implement it.

2. Change NSPR's _darwin.cfg to say

  #if defined(__ppc__) || defined(ppc)
  #undef IS_LITTLE_ENDIAN
  #define  IS_BIG_ENDIAN 1
  #else
  #undef IS_BIG_ENDIAN
  #define  IS_LITTLE_ENDIAN 1
  #endif

and compile jscpucfg.c with -DCROSS_COMPILE -Dppc.  Note that
we can't use -D__ppc__ because __ppc__ is a compiler predefined
macro and defining it may confuse <stdio.h> and <stdlib.h>,
which jscpucfg.c includes.  I don't like this approach because
NSPR's _darwin.cfg will need to keep the define(ppc) test until
JS drops support for Mac OS X PowerPC.

3. In _darwin.cfg, change __i386__ back to i386 (attachment
350272 [review]).  I don't like this approach because it will undo the
fix for bug 417044.

Comment 78

[Wan-Teh Chang]

Attachment: Revert back to NSPR_4_7_1_RTM [Checkin: Comment 87]

Samuel: I can check this in so that this won't block 1.9.0.5
(if we can't get a timely review for my JS patch).

Comment 79

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

> (In reply to comment #66)
> > Smokey, please apply this patch to NSPR_4_7_3_RTM and do a
> > *full* clobber build.  (Code outside nsprpub also needs to
> > be recompiled.)
> 
> I'll start this now.

http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081127-fullclb-NSPR473-patched.dmg

I'll back out the patch from comment 66 and try the patch to JS (comment 77) next.

Comment 80

[David Ryeburn]

(In reply to comment #79)
This seems to work OK. I tried the NY Times again, including pausing videos. I'll use it until your next version arrives, or it breaks, whichever comes first.

Comment 81

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #77)
> Created an attachment (id=350389) [details]
> Proposed patch for JS
> 
> Here is the patch I propose for JS.  Because of the difficulty
> of testing for endianness of the target processor when we cross
> compile, jscpucfg.c can generate code that tests predefined
> macros for processor architectures.
> 
> Smokey, could you test this patch with NSPR_4_7_3_RTM in a
> full clobber build?

This build (stock NSPR_4_7_3_RTM plus the patch to JS from comment 77) is http://www.ardisson.org/smokey/moz/Camino-Cm2-M1.9-20081127-fullclb-NSPR473-js-patched.dmg

Comment 82

[David Ryeburn]

(In reply to comment 81)

This one seems to work fine too. Does this indicate that the problem is a conflict between the original NSPR 473 and Javascript files, which can be fixed at either the NSPR end or the Javascript end? Or am I misunderstanding what is going on?

Comment 83

[Wan-Teh Chang]

Comment on attachment 350361 [details]
compiler commands from (last night's) jsdtoa.c compile

Thank you for generating and testing all the experimental builds.
With the incorrectly generated jsautocfg.h for ppc (attachment 350356 [details])
and the latest test results, we know the problem is indeed what I
suspected in comment 69 and comment 70.

This bug is caused by JS's dependence on a property of the NSPR
header _darwin.cfg that changed in NSPR 4.7.3.  Although _darwin.cfg
is a public header, this property (the macro that _darwin.cfg tests
to determine the endianness of the processor) is an implementation
detail.  So it is better to fix this bug by eliminating JS's dependence
on this implementation detail, even though we could also fix this bug
by changing NSPR (restoring the property that JS depends on).

Comment 84

[Wan-Teh Chang]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

Mark, I'm asking you to review this patch first because
you wrote the relevant code in js/src/Makefile.in for bug
322578.

cls, you added CROSS_COMPILE support to js/src/jscpucfg.c
in rev. 3.10, so I'm also asking you to review this patch.

This bug report is very long, so let me summarize it for
you.  The bug is that after I changed the test for CPU
endianness from defined(i386) to defined(__i386__) in
NSPR's _darwin.cfg, Universal binaries built on x86 started
to crash when running on ppc.  I explained the problem in
comment 64, comment 69, and comment 70, and suggested some
alternative solutions in comment 77.

Comment 85

[Wan-Teh Chang]

Comment on attachment 350390 [details] [diff] [review]
Revert back to NSPR_4_7_1_RTM
[Checkin: Comment 87]

I certainly would like to fix the jscpucfg.c problem, but
in the interest of time, we can revert to NSPR_4_7_1_RTM
for 1.9.0.5.

Comment 86

[Samuel Sidler (old account; do not CC)]

Comment on attachment 350390 [details] [diff] [review]
Revert back to NSPR_4_7_1_RTM
[Checkin: Comment 87]

r=me.

Approved for 1.9.0.5. a=ss for release-drivers.

Please land this in CVS as soon as possible so we can watch nightlies and ensure this is fixed. We'll take the NSPR upgrade for 1.9.0.6, in conjunction with the other patch in this bug after it gets reviews.

Comment 87

[Daniel Veditz [:dveditz]]

I checked in the NSPR backout. I think this is a 1.9.0-only crash, but we'll want the js build changes later when bug 463073 lands again (and on trunk) so I'll leave this bug open.

Comment 88

[Brian Crowder]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

This looks okay to me...  Might be nice if Jim Blandy could have a look, too (with an eye toward problems like this on trunk?)

Comment 89

[Wan-Teh Chang]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

Jim, could you please review this patch, too?

It turns out that the jscpucfg.c CROSS_COMPILE problem is a
known bug: bug 269538.  This bug is an instance of that bug.

Comment 90

[Mark Mentovai]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

This seems fine.  You could also have checked the compiler-predefined macros __LITTLE_ENDIAN__ or __BIG_ENDIAN__.

Comment 91

[Jim Blandy :jimb]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

My comments are identical to comment 90.  It'd be nice if SpiderMonkey could do all this with autoconf...

Comment 92

[Wan-Teh Chang]

Mark, Jim, did you mean the new code I added to jscpucfg.c
should look like this?

+    printf("#if defined(__LITTLE_ENDIAN__)\n");
+    printf("#undef IS_BIG_ENDIAN\n");
+    printf("#define  IS_LITTLE_ENDIAN 1\n");
+    printf("#else\n");
+    printf("#undef IS_LITTLE_ENDIAN\n");
+    printf("#define  IS_BIG_ENDIAN 1\n");
+    printf("#endif\n\n");

Comment 94

[Al Billings [:abillings - ex-MoCo]]

Verified that this is fixed with 1.9.0.5 official mac builds.

Comment 95

[Wan-Teh Chang]

On the Firefox trunk in mozilla-central, jscpucfg for ppc is compiled
and jsautocfg.h for ppc is generated with these command lines:

g++-4.0 -DMDCPUCFG=\"md/_darwin.cfg\" -gstabs -gfull -save-temps  -DAVMPLUS_UNIX -DOSTYPE=\"Darwin9.2.0\" -DOSARCH=Darwin -DEXPORT_JS_API  -DJS_USE_SAFE_ARENA -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include/nspr -o jscpucfg /builds/moz2_slave/mozilla-central-macosx-nightly/build/js/src/jscpucfg.cpp
./jscpucfg > jsautocfg.tmp
mv jsautocfg.tmp jsautocfg.h

Note that the g++-4.0 command line doesn't have -DCROSS_COMPILE or
-include ./mozilla-config.h, so the CROSS_COMPILE macro is *not*
defined when we compile jscpucfg.cpp for ppc.  This means my proposed
patch for JS (attachment 350389 [details] [diff] [review]) has no effect for mozilla-central.

I also verified that the jsautocpu.h file for ppc is incorrect; it
defines IS_LITTLE_ENDIAN.  Because CROSS_COMPILE is not defined, I
don't know how to fix it.  I also don't understand why it doesn't
result in a crash when running on ppc.

Comment 96

[Wan-Teh Chang]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

Requesting 1.9.0.6 approval.

As I explained in comment 95, this patch has no effect for 1.9.1,
but for some reason 1.9.1 doesn't seem to be affected by this
bug.

Comment 97

[Samuel Sidler (old account; do not CC)]

If this patch is the right thing to do, we should probably take it for 1.9.1 and trunk as well, no? I'd hate to get out of sync and have it bite us later. (And I like baking time somewhere other than the stable branch.)

Comment 98

[Wan-Teh Chang]

I filed bug 467677 for the 1.9.1/mozilla-central problem I described
in comment 95.

Comment 99

[Jim Blandy :jimb]

(In reply to comment #92)
> Mark, Jim, did you mean the new code I added to jscpucfg.c
> should look like this?

Yes, that's what seemed right to me.

Comment 100

[Wan-Teh Chang]

Attachment: Proposed patch for JS (trunk) [Checkin: Comment 105]

Comment 101

[Wan-Teh Chang]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

Jim, could you check this patch in on the trunk (mozilla-central) if it
looks good?

This patch is a no-op on the trunk until bug 467677 is fixed.  Then
it will generate the following code in jsautocfg.h when we cross compile
for Mac universal binaries:

#ifdef __LITTLE_ENDIAN__
#define IS_LITTLE_ENDIAN 1
#undef  IS_BIG_ENDIAN
#else
#undef  IS_LITTLE_ENDIAN
#define IS_BIG_ENDIAN 1
#endif

Note that I switched to __LITTLE_ENDIAN__ on your suggestion.

Comment 102

[Jim Blandy :jimb]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

Looks good.  I can't land patches; you'll have to find a different patch buddy.

One thought: although $host != $target, building PPC Mozilla on i386 Darwin isn't cross compiling in some important senses.  1) You can run the executables, so dynamic tests like jsautocfg in its normal mode are fine.  2) You don't need a special toolchain.  So I think there's some opportunity for further simplification in the whole universal build process.

Comment 103

[Wan-Teh Chang]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

Brian, could you check in this patch on the trunk (mozilla-central)
for me?  Thanks.

Jim, what you said is correct.  There is a corner case: you can
build a universal binary on ppc, or cross compile for x86 on ppc.
In that case, it's still cross compiling in the conventional sense.
I don't think anyone is cross compiling in that direction now.

Comment 104

[Brian Crowder]

I may not get to it today, so hopefully someone else can land if I don't.

Comment 105

[Serge Gautherie (:sgautherie)]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

http://hg.mozilla.org/mozilla-central/rev/7604a342ad4b

Comment 106

[Samuel Sidler (old account; do not CC)]

Resolved FIXEd per comment 105. This needs 1.9.1 landing though.

Comment 107

[Samuel Sidler (old account; do not CC)]

Comment on attachment 350389 [details] [diff] [review]
Proposed patch for JS

1.9.1 drivers: We'd like this to land on 1.9.1 to get sufficient baking and ensure it does not cause any side effects before taking in 1.9.0.6 so we can upgrade NSPR in 1.9.0.6.

Comment 108

[Samuel Sidler (old account; do not CC)]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

Whoops. Wrong patch.

1.9.1 drivers: See comment 107.

Comment 109

[Wan-Teh Chang]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

The baking of this patch on the tip officially started last
night (2008-12-12).  Although this patch was checked in on
2008-12-06, it was not activated until I checked in the patch
in bug 467677 comment 14.

Comment 110

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 351333 [details] [diff] [review]
Proposed patch for JS (trunk)
[Checkin: Comment 105]

a191=beltzner let's get this into nightlies and watch for regressions while everyone's still around this week

Comment 111

[Wan-Teh Chang]

I pushed changeset 595a2575da97 to mozilla-1.9.1.
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/595a2575da97

Comment 112

[Wan-Teh Chang]

Attachment: Proposed patch for JS and upgrade to NSPR_4_7_3_RTM (Firefox 3.0 CVS HEAD)

Requesting 1.9.0.6 approval.  This patch has been pushed to mozilla-central and
mozilla-1.9.1.  Please see attachment 350389 [details] [diff] [review] (an older version of the patch) for
the original reviews.

Comment 113

[Daniel Veditz [:dveditz]]

Comment on attachment 353918 [details] [diff] [review]
Proposed patch for JS and upgrade to NSPR_4_7_3_RTM (Firefox 3.0 CVS HEAD) 

Approved for 1.9.0.6, a=dveditz for release-drivers.

wtc: mozilla/js is a restriced partition, you'll have to get someone to check in for you

Comment 114

[Wan-Teh Chang]

Comment on attachment 353918 [details] [diff] [review]
Proposed patch for JS and upgrade to NSPR_4_7_3_RTM (Firefox 3.0 CVS HEAD) 

Could someone with CVS commit access to mozilla/js please check
in this patch for me?  Here is the CVS commit comment I wrote:

Bug 466531: Upgraded to NSPR 4.7.3.  Added a Mac OS X workaround for
the bug that jscpucfg.c doesn't define the correct endianness macro
for cross compilation to an OS with multiple CPU architectures.  Patch
by Wan-Teh Chang <[e-mail]>.  r=mark,crowder,jim.  Approved for
1.9.0.6, a=dveditz for release-drivers.

Comment 115

[:Gavin Sharp [email: [e-mail]]]

Checked in on the CVS HEAD for 1.9.0.6:
mozilla/client.mk 	1.389
mozilla/js/src/Makefile.in 	3.126
mozilla/js/src/jscpucfg.c 	3.28

Comment 116

[Samuel Sidler (old account; do not CC)]

Can someone on PPC verify they still don't crash using tomorrow's build? Chris, I'm looking at you and your sweet dial-up. ;)

(Please use the verified1.9.0.6 if you do not crash.)

Comment 117

[Chris Lawson (gone)]

With Camino 2008122600, I'm unable to repro the crash using the STR in comment 0. Sounds VERIFIED to me.

Comment 118

[Samuel Sidler (old account; do not CC)]

Thanks Chris!