Closed Bug 1311462 Opened 5 years ago Closed 5 years ago
Symbol dumping failing for XUL on Firefox Beta 8 builds
[Tracking Requested - why for this release]: Need symbols for Mac during the run up to final release +++ This bug was initially created as a clone of Bug #1301751 +++ Beta 8 Mac builds seem to be suffering from the same issue seen in the cloned bug. See http://bit.ly/2eFhoe9
From my comment on that bug: I looked at the 50.0b8 OS X build log: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=1815769 and I didn't see any evidence of this bug happening. There are no llvm-dsymutil crash stacks. There are a bunch of warnings like: 09:29:55 INFO - warning: no debug symbols in executable (-arch x86_64)
Summary: llvm-dsymutil crashing on XUL - Firefox Beta 8 builds → Symbol dumping failing for XUL on Firefox Beta 8 builds
Ted: Who would be a good person to help us investigate this?
I don't see anything in the changes between b7 and b8 that look suspicious: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?rev=%27FIREFOX_50_0b7_RELEASE::FIREFOX_50_0b8_RELEASE%27
Assignee: nobody → ted
Bleh, that query wasn't quite right. This is close: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_50_0b7_RELEASE&tochange=FIREFOX_50_0b8_RELEASE
Looking at the opt builds on the latest pushes they all seem to be displaying this same failure...
Seems to be broken back at the b7 builds too: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&fromchange=c216304c306bda09e8b8766f692885fa378ddb3c&selectedJob=1796384 I'll keep looking.
I grepped all the build logs, this appears to have been broken all the way back to the 50.0b1 build.
Here's the beta 1 tag: https://hg.mozilla.org/releases/mozilla-beta/rev/FIREFOX_50_0b1_RELEASE The indexed task for its artifacts: https://tools.taskcluster.net/task-inspector/#NLkcFlKYSem3H9GbK99a4w/ Its build log: https://public-artifacts.taskcluster.net/Fc_7lPnlTKWHd2DE7yqYGg/0/public/build/log_raw.log
With RyanVM's help I looked at the last Aurora build from the 50 cycle: https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&tochange=3fdfaaed6e0fad2d81fc28e49903ff6f0a43b12e&fromchange=2d4867f1d03306471fec3f679896365845985f69&selectedJob=3583159 That build does not display this failure mode, so it's something specific to beta.
Okay, I think this is a regression from bug 1244446: https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b One major difference between the aurora and beta configs is that in nightly/aurora we build with --disable-install-strip: https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/nightly#3 and on beta and release we don't: https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/beta That means we strip during packaging on beta. Prior to that change, we would do a special stage-package after the build, which explicitly passed PKG_SKIP_STRIP=1: https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b#l1.35 With that change, we just do the normal stage-package, and so on beta we stage dist/universal and presumably strip it along the way! We didn't notice this on nightly/aurora because of --disable-install-strip.
glandium: we'll need to figure out some workaround here until we can fix symbol dumping to be better. The best thing I can come up with right now is "always make universal builds skip stripping during packaging, make symbolstore.py strip the binaries after dumping symbols". We could do the stripping in symbolstore.py here: https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/toolkit/crashreporter/tools/symbolstore.py#989
I filed bug 1314137 on making sure this errors in the future.
AFAICT, before bug 1244446, mac universal builds on beta were, in fact, not much different from a --disable-install-strip build. I'd say: add --disable-install-strip to the release mozconfig, and don't care about stripping after the fact. We weren't caring about doing it before, I don't see a reason to care more just to fix this issue. My interpretation of the situation before bug 1255556 could be wrong, though.
> My interpretation of the situation before bug 1255556 could be wrong, though. bug 1244446, not 1255556.
Comment on attachment 8806278 [details] Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. https://reviewboard.mozilla.org/r/89774/#review89212 Oh, that is a nice solution! If the try build looks good I'll land this.
Attachment #8806278 - Flags: review?(ted) → review+
I checked out the build log and the resulting symbols, they both look good. One fiddly thing in the build log is a bunch of warnings like: 03:54:58 INFO - warning: cannot open debug object "/builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/js/src/gdb/../libjs_static.a(decContext.o)": No such file or directory We must be linking some of the binaries under js/src with relative paths to the static libraries, so dsymutil can't find the objects. It's not a big deal, it'll only impact the i386 symbols for those binaries, which probably aren't used anywhere anyway. (Also this will all go away when we disable universal builds.)
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/7d59e316811b Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. r=ted
Comment on attachment 8806278 [details] Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. Approval Request Comment [Feature/regressing bug #]: bug 1244446 caused this regression, which didn't show up until beta due to differing build config [User impact if declined]: No symbols for OS X crash reports [Describe test coverage new/current, TreeHerder]: None, but this is a build-config only patch. [Risks and why]: It's possible this could break something in symbol dumping, but that's already broken on OS X on beta. The patch is very targeted. [String/UUID change made/needed]: N/A
Comment on attachment 8806278 [details] Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. This is a blocking issue for 50 release, Aurora51+, Beta50+
You need to log in before you can comment on or make changes to this bug.