Symbol dumping failing for XUL on Firefox Beta 8 builds

RESOLVED FIXED in Firefox 50

Status

defect
P1
normal
RESOLVED FIXED
3 years ago
Last year

People

(Reporter: marcia, Assigned: glandium)

Tracking

50 Branch
mozilla52
Dependency tree / graph

Firefox Tracking Flags

(firefox50blocking fixed, firefox51 fixed, firefox52 fixed)

Details

Attachments

(1 attachment)

[Tracking Requested - why for this release]: Need symbols for Mac during the run up to final release

+++ This bug was initially created as a clone of Bug #1301751 +++

Beta 8 Mac builds seem to be suffering from the same issue seen in the cloned bug. See 

http://bit.ly/2eFhoe9
From my comment on that bug:


I looked at the 50.0b8 OS X build log:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=1815769

and I didn't see any evidence of this bug happening. There are no llvm-dsymutil crash stacks.

There are a bunch of warnings like:
09:29:55     INFO -  warning: no debug symbols in executable (-arch x86_64)
Summary: llvm-dsymutil crashing on XUL - Firefox Beta 8 builds → Symbol dumping failing for XUL on Firefox Beta 8 builds
Ted: Who would be a good person to help us investigate this?
Flags: needinfo?(ted)
I don't see anything in the changes between b7 and b8 that look suspicious:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?rev=%27FIREFOX_50_0b7_RELEASE::FIREFOX_50_0b8_RELEASE%27
Assignee: nobody → ted
Flags: needinfo?(ted)
Looking at the opt builds on the latest pushes they all seem to be displaying this same failure...
I grepped all the build logs, this appears to have been broken all the way back to the 50.0b1 build.
With RyanVM's help I looked at the last Aurora build from the 50 cycle:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&tochange=3fdfaaed6e0fad2d81fc28e49903ff6f0a43b12e&fromchange=2d4867f1d03306471fec3f679896365845985f69&selectedJob=3583159

That build does not display this failure mode, so it's something specific to beta.
Okay, I think this is a regression from bug 1244446:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b

One major difference between the aurora and beta configs is that in nightly/aurora we build with --disable-install-strip:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/nightly#3

and on beta and release we don't:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/beta

That means we strip during packaging on beta. Prior to that change, we would do a special stage-package after the build, which explicitly passed PKG_SKIP_STRIP=1:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b#l1.35

With that change, we just do the normal stage-package, and so on beta we stage dist/universal and presumably strip it along the way! We didn't notice this on nightly/aurora because of --disable-install-strip.
Blocks: 1244446
glandium: we'll need to figure out some workaround here until we can fix symbol dumping to be better. The best thing I can come up with right now is "always make universal builds skip stripping during packaging, make symbolstore.py strip the binaries after dumping symbols".

We could do the stripping in symbolstore.py here:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/toolkit/crashreporter/tools/symbolstore.py#989
Flags: needinfo?(mh+mozilla)
This is a blocking issue for Fx50 release.
I filed bug 1314137 on making sure this errors in the future.
Assignee

Comment 14

3 years ago
AFAICT, before bug 1244446, mac universal builds on beta were, in fact, not much different from a --disable-install-strip build. I'd say: add --disable-install-strip to the release mozconfig, and don't care about stripping after the fact. We weren't caring about doing it before, I don't see a reason to care more just to fix this issue. My interpretation of the situation before bug 1255556 could be wrong, though.
Flags: needinfo?(mh+mozilla)
Assignee

Comment 15

3 years ago
> My interpretation of the situation before bug 1255556 could be wrong, though.

bug 1244446, not 1255556.

Comment 18

3 years ago
mozreview-review
Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

https://reviewboard.mozilla.org/r/89774/#review89212

Oh, that is a nice solution! If the try build looks good I'll land this.
Attachment #8806278 - Flags: review?(ted) → review+
Assignee: ted → mh+mozilla
I checked out the build log and the resulting symbols, they both look good. One fiddly thing in the build log is a bunch of warnings like:
03:54:58     INFO -  warning: cannot open debug object "/builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/js/src/gdb/../libjs_static.a(decContext.o)": No such file or directory

We must be linking some of the binaries under js/src with relative paths to the static libraries, so dsymutil can't find the objects. It's not a big deal, it'll only impact the i386 symbols for those binaries, which probably aren't used anywhere anyway. (Also this will all go away when we disable universal builds.)

Comment 20

3 years ago
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7d59e316811b
Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. r=ted
Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

Approval Request Comment
[Feature/regressing bug #]: bug 1244446 caused this regression, which didn't show up until beta due to differing build config
[User impact if declined]: No symbols for OS X crash reports
[Describe test coverage new/current, TreeHerder]: None, but this is a build-config only patch.
[Risks and why]: It's possible this could break something in symbol dumping, but that's already broken on OS X on beta. The patch is very targeted.
[String/UUID change made/needed]: N/A
Attachment #8806278 - Flags: approval-mozilla-beta?
Attachment #8806278 - Flags: approval-mozilla-aurora?
Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

This is a blocking issue for 50 release, Aurora51+, Beta50+
Attachment #8806278 - Flags: approval-mozilla-beta?
Attachment #8806278 - Flags: approval-mozilla-beta+
Attachment #8806278 - Flags: approval-mozilla-aurora?
Attachment #8806278 - Flags: approval-mozilla-aurora+

Comment 24

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7d59e316811b
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52

Updated

Last year
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.