1311462 - Symbol dumping failing for XUL on Firefox Beta 8 builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Marcia Knous [:marcia]]

[Tracking Requested - why for this release]: Need symbols for Mac during the run up to final release

+++ This bug was initially created as a clone of Bug #1301751 +++

Beta 8 Mac builds seem to be suffering from the same issue seen in the cloned bug. See 

http://bit.ly/2eFhoe9

Comment 1

[(not currently active) Ted Mielczarek]

From my comment on that bug:


I looked at the 50.0b8 OS X build log:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=1815769

and I didn't see any evidence of this bug happening. There are no llvm-dsymutil crash stacks.

There are a bunch of warnings like:
09:29:55     INFO -  warning: no debug symbols in executable (-arch x86_64)

Comment 2

[Marcia Knous [:marcia]]

Ted: Who would be a good person to help us investigate this?

Comment 3

[(not currently active) Ted Mielczarek]

I don't see anything in the changes between b7 and b8 that look suspicious:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?rev=%27FIREFOX_50_0b7_RELEASE::FIREFOX_50_0b8_RELEASE%27

Comment 4

[(not currently active) Ted Mielczarek]

Bleh, that query wasn't quite right. This is close:
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_50_0b7_RELEASE&tochange=FIREFOX_50_0b8_RELEASE

Comment 5

[(not currently active) Ted Mielczarek]

Looking at the opt builds on the latest pushes they all seem to be displaying this same failure...

Comment 6

[(not currently active) Ted Mielczarek]

Seems to be broken back at the b7 builds too:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&fromchange=c216304c306bda09e8b8766f692885fa378ddb3c&selectedJob=1796384

I'll keep looking.

Comment 7

[(not currently active) Ted Mielczarek]

I grepped all the build logs, this appears to have been broken all the way back to the 50.0b1 build.

Comment 8

[(not currently active) Ted Mielczarek]

Here's the beta 1 tag:
https://hg.mozilla.org/releases/mozilla-beta/rev/FIREFOX_50_0b1_RELEASE

The indexed task for its artifacts:
https://tools.taskcluster.net/task-inspector/#NLkcFlKYSem3H9GbK99a4w/

Its build log:
https://public-artifacts.taskcluster.net/Fc_7lPnlTKWHd2DE7yqYGg/0/public/build/log_raw.log

Comment 9

[(not currently active) Ted Mielczarek]

With RyanVM's help I looked at the last Aurora build from the 50 cycle:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&tochange=3fdfaaed6e0fad2d81fc28e49903ff6f0a43b12e&fromchange=2d4867f1d03306471fec3f679896365845985f69&selectedJob=3583159

That build does not display this failure mode, so it's something specific to beta.

Comment 10

[(not currently active) Ted Mielczarek]

Okay, I think this is a regression from bug 1244446:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b

One major difference between the aurora and beta configs is that in nightly/aurora we build with --disable-install-strip:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/nightly#3

and on beta and release we don't:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/browser/config/mozconfigs/macosx-universal/beta

That means we strip during packaging on beta. Prior to that change, we would do a special stage-package after the build, which explicitly passed PKG_SKIP_STRIP=1:
https://hg.mozilla.org/mozilla-central/rev/e1bd07ec0b5b#l1.35

With that change, we just do the normal stage-package, and so on beta we stage dist/universal and presumably strip it along the way! We didn't notice this on nightly/aurora because of --disable-install-strip.

Comment 11

[(not currently active) Ted Mielczarek]

glandium: we'll need to figure out some workaround here until we can fix symbol dumping to be better. The best thing I can come up with right now is "always make universal builds skip stripping during packaging, make symbolstore.py strip the binaries after dumping symbols".

We could do the stripping in symbolstore.py here:
https://dxr.mozilla.org/mozilla-central/rev/e3279760cd977aac30bd9e8032d3ee71f55d2a67/toolkit/crashreporter/tools/symbolstore.py#989

Comment 12

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

This is a blocking issue for Fx50 release.

Comment 13

[(not currently active) Ted Mielczarek]

I filed bug 1314137 on making sure this errors in the future.

Comment 14

[Mike Hommey [:glandium]]

AFAICT, before bug 1244446, mac universal builds on beta were, in fact, not much different from a --disable-install-strip build. I'd say: add --disable-install-strip to the release mozconfig, and don't care about stripping after the fact. We weren't caring about doing it before, I don't see a reason to care more just to fix this issue. My interpretation of the situation before bug 1255556 could be wrong, though.

Comment 15

[Mike Hommey [:glandium]]

> My interpretation of the situation before bug 1255556 could be wrong, though.

bug 1244446, not 1255556.

Comment 16

[Mike Hommey [:glandium]]

Attachment: Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

Review commit: https://reviewboard.mozilla.org/r/89774/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/89774/

Comment 17

[Mike Hommey [:glandium]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=abbe85b60688c69188b875b0a924149146599a19

Comment 18

[(not currently active) Ted Mielczarek]

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

https://reviewboard.mozilla.org/r/89774/#review89212

Oh, that is a nice solution! If the try build looks good I'll land this.

Comment 19

[(not currently active) Ted Mielczarek]

I checked out the build log and the resulting symbols, they both look good. One fiddly thing in the build log is a bunch of warnings like:
03:54:58     INFO -  warning: cannot open debug object "/builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/js/src/gdb/../libjs_static.a(decContext.o)": No such file or directory

We must be linking some of the binaries under js/src with relative paths to the static libraries, so dsymutil can't find the objects. It's not a big deal, it'll only impact the i386 symbols for those binaries, which probably aren't used anywhere anyway. (Also this will all go away when we disable universal builds.)

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7d59e316811b
Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. r=ted

Comment 21

[(not currently active) Ted Mielczarek]

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

Approval Request Comment
[Feature/regressing bug #]: bug 1244446 caused this regression, which didn't show up until beta due to differing build config
[User impact if declined]: No symbols for OS X crash reports
[Describe test coverage new/current, TreeHerder]: None, but this is a build-config only patch.
[Risks and why]: It's possible this could break something in symbol dumping, but that's already broken on OS X on beta. The patch is very targeted.
[String/UUID change made/needed]: N/A

Comment 22

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8806278 [details]
Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds.

This is a blocking issue for 50 release, Aurora51+, Beta50+

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/45118388b658d19e06fe553cc41654eca677c64c
https://hg.mozilla.org/releases/mozilla-beta/rev/b36718709777b50df48b800c7f05674cfa9e2701
https://hg.mozilla.org/releases/mozilla-release/rev/b36718709777b50df48b800c7f05674cfa9e2701

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7d59e316811b