Closed Bug 448616 Opened 15 years ago Closed 15 years ago

no useful breakpad symbols for OS X builds

Categories

(Release Engineering :: General, defect, P1)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: ted, Assigned: bhearsum)

Details

(Keywords: verified1.9.1)

Attachments

(2 files, 1 obsolete file)

Output on buildsymbols attempt
2.98 KB, text/plain
Details
[checked in] symbols before packaging before snippets
1.23 KB, patch
ted
: review+
Details | Diff | Splinter Review 
Apparently we haven't had useful breakpad symbols for our OS X builds for quite some time. bhearsum and I are not sure if this ever actually worked, although given the gyrations we went through, I'm not sure how that would have happened.

We shipped 3.1a1 without useful symbols, and we don't have useful symbols for any build from the past 30 days.

3.0.x builds are fine.
I'm going to try building mozilla-central on one of our x86 10.4 machines tomorrow (if I have time). This should tell us if the problem is with the Xcode change. Not sure of a plan of action after that...
Similar to bug 411171 (even though that's cvs/1.9.0) or something different ? Are we talking a failure when calling symbolstore or getting garbage out ?
no, dumpsyms is running fine, but there appears to be no symbol info in the object files. 
Specifically, the ppc objects have a complete set of symbols. The i386 ones only have PUBLIC symbols.
bhearsum and I think this broke when we switched to OSX 10.5. We don't have enough history at this point to prove otherwise, but it's been broken for at least the past 30 days, and I don't see any differences in build setup that would account for it, so I conclude that it has been broken since we setup these machines.

As I see it, we have two options:
1) Get bug 421534 fixed and switch to DWARF
2) Revert these build machines to OS X 10.4 to match the 1.9 build machines

Definitely need to block 1.9.1 on this, either way.
Flags: blocking1.9.1?
(In reply to comment #5)
> Definitely need to block 1.9.1 on this, either way.

Agree with blocking 1.9.1 on this. Should this block FF3.1b1?
Going back to 10.4 requires using using an older XCode too, 2.5 IIRC. Perhaps we can go the other way and update our 10.5/XCode 3 to Xcode 3.1 ? My kingdom for an Apple changelog ...
I have XCode 3.1 here, I can do a build and see if this appears to work. (Although I guess the best test would be doing it on a spare build machine, if we have one of those.)

This is what I got when I did a build on bm-xserve-unittest-01.b.m.o (it's in /Users/cltbld/moz2).  This is with XCode 3.1 installed.

Happy to help on this if someone will tell me what to look for.
sorry about that, it was a python issue.

now I run make -C browser buildsymbols and I do in fact get symbols:

http://avnerd.tv/sharedFiles/crashreporter-symbols-firefox-3.1a2pre-Darwin-2008080515.zip


Lukas, are those i386 or ppc symbols? (Is your Mac Intel or PPC?) If those are i386 I would be very interested in trying this on a build Mac.
Well, duh:  MODULE mac x86 4BF8230617D3F9AF2ACE439C154C32FB0 libbrowsercomps.dylib


We should try out Xcode 3.1 on one of our Macs.
Assignee: nobody → lukasblakk
Status: NEW → ASSIGNED
Summary: no useful breakpad symbols for OS X builds → no useful breakpad symbols for OS X builds (update mac build machines to XCode 3.1)
Lukas volunteering to try Xcode 3.1 on bm-xserve18. I've pulled it out of the production pool while she does this._ZZZZs/
I examined the build Lukas did and the i386 symbols are intact. We should upgraded the rest of our Macs ASAP. I'm testing right now to see if we can also drop -save-temps by upgrading to Xcode 3.1.
ftr, the build worked without -save-temps. Upgrading to Xcode 3.1 is going to save us a ton of disk space on Mac.
That's great news, but before we rush to upgrade the boxes has anyone tried running one of those builds ? I think it would also be worth polling developers (maybe a newsgroup post) to see if there is any detrimental fallout from upgrading XCcode to 3.1
FWIW, bhearsum notes that the default gcc with XCode 3.1 is still 4.0.1. It provides gcc 4.2, but that's not the default, so there should be very little churn here aside from bug fixes. We'll continue using the 10.4 SDK, as well.

copied the objdir for i386 over to bm-stage-osx-01 which runs 10.4.11 and got this output when I tried to run the build:

bm-stage-osx-01:~/Desktop cltbld$ i386/dist/Minefield.app/Contents/MacOS/firefox-bin 
dyld: Library not loaded: @executable_path/XUL
  Referenced from: /Users/cltbld/Desktop/i386/dist/Minefield.app/Contents/MacOS/firefox-bin
  Reason: image not found
Trace/BPT trap

Did "make package" on the i386 objdir and copied the .dmg over to bm-stage-osx-01, installed Minefield and started it up.  All seems well.  

Steps to update to 3.1 were pretty straightforward: 
1) copy the xcode31.dmg (currently on bm-xserve18 in /Users/cltbld)
2) install it


Yeah, this blocks 3.1b1 IMO.
Flags: blocking1.9.1? → blocking1.9.1+
bm-xserve16
bm-xserve17
bm-xserve18
bm-xserve19

are all up to date with xcode 3.1 now
Forced a nightly, which finished at 13:30 on August 8, 2008 - someone please check the symbols and see if they are working as expected.
No working symbols on that build :(. I compared the mozconfigs that you used on bm-xserve18, and the ones the nightly was produced with. The only difference is you didn't use -save-temps. I'm going to disable this in the official configs are re-spin the nightly.
(we had planned to remove -save-temps anyways)
Removing -save-temps seems to have broken clobber builds. Not sure why this is happening now and not in our testing. Going to re-enable it for the time being to make sure dep builds don't break.
Priority: -- → P1
Assignee: lukasblakk → bhearsum
Status: ASSIGNED → NEW
There is strong indications that this is happening because we do 'make package' before 'make buildsymbols'. I'm doing tests right now to confirm this.
I've confirmed it: 'make package' before 'make buildsymbols' breaks symbols for universal builds (and maybe non-universal ones). This is because 'make package' strips dist/universal objects. When 'buildsymbols' comes along and picks them up there's nothing of value there.

Patches incoming.
I've moved packaging+upload right to the end to be in line with the order Tinderbox client does things, just in case another weird thing like this comes up.

I had to duplicate the 'SetMozillaBuildProperties' step since the mar generation was moved to happen right before snippet generation.

I'm currently testing this patch on staging.
Attachment #335911 - Flags: review?(ted.mielczarek)
Comment on attachment 335911 [details] [diff] [review]
do symbol build/upload before packaging

yes, please!

Thanks for the detective work!
Attachment #335911 - Flags: review?(ted.mielczarek) → review+
I'm really busy with releases today, I still need to make sure that patch doesn't regress anything. The good news is that we'll have symbols for 3.1a2
Great, I've changed the Thunderbird buildbots accordingly

http://hg.mozilla.org/build/buildbot-configs/rev/281

changeset:   281:26a1b5d2aafa
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Thu Sep 04 10:35:01 2008 -0400
summary:     Bug 448616. Move the buildsymbols generation target early, before we run make dist and clobber all usefull symbols.
Is this fixed now?
no, not yet. i'm testing a solution for our nightlies in staging right now (in between doing releases). hope to have it in production by the end of the week.
Attachment #335911 - Attachment is obsolete: true
Turns out that we have to do symbols *before* packaging on Mac, because 'make package' strips the symbols. Also, we cannot create a complete mar before running 'make package' on Linux, because the required directory will not exist.

This patch has been fully tested on all 3 platforms in staging, I would be shocked if it breaks anything.
Attachment #337099 - Flags: review?(ted.mielczarek)
Comment on attachment 337099 [details] [diff] [review]
[checked in] symbols before packaging before snippets

r=me, but did you really mean to take out that check for self.nightly?
Attachment #337099 - Flags: review?(ted.mielczarek) → review+
Summary: no useful breakpad symbols for OS X builds (update mac build machines to XCode 3.1) → no useful breakpad symbols for OS X builds
(In reply to comment #35)
> (From update of attachment 337099 [details] [diff] [review])
> r=me, but did you really mean to take out that check for self.nightly?

Yeah - there wasn't a real good reason for that business logic being there in the first place.
Comment on attachment 337099 [details] [diff] [review]
[checked in] symbols before packaging before snippets

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.13; previous revision: 1.12
done
Attachment #337099 - Attachment description: symbols before packaging before snippets → [checked in] symbols before packaging before snippets
OK, here's a crash I just had with the latest OS X nightly: http://crash-stats.mozilla.com/report/index/a846ad4a-7db2-11dd-b5bc-001cc4e2bf68

The functions appear to be correct. No line numbers in the Frames/Modules due to bug 454198, but they *are* in the raw dump.

I'm pretty sure we can call this bug fixed.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Yes, the log looks accurate to me.

But I assume you used the "crashme" extension to cause the crash, and
that your log doesn't represent a real bug.  (I wasn't previously
familiar with this extension.)
(In reply to comment #38) 
> I'm pretty sure we can call this bug fixed.

Yeah, that's fine for a really long time now => verified
Status: RESOLVED → VERIFIED
Fixed 1.9.1 as per bhearsum on IRC
Keywords: fixed1.9.1
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.