495030 - Try server Mac builds have bogus symbols for libxul

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Henri Sivonen (:hsivonen)]

Steps to reproduce:
 1) Download https://build.mozilla.org/tryserver-builds/[e-mail]-try-56694b17d6a/try-56694b17d6a-macosx.dmg
 2) Try to profile it in Shark.

Actual results:
The stacks are utterly bogus for library "XUL". They seem sane for all the other libs.

Expected results:
Expected the OS to be able to retrieve correct stacks for libxul as is the case with locally-built opt builds.

Additional info:
The bogosity of the symbols also affects Crash Reporter and Valgrind. (Oh how I wish I had realized earlier that the symbols are overall bogus. I've been mislead by bogus crash stacks as early as March, but I didn't realize the fault was in the builds. I thought it was an isolated issue with a particular crash.)

Marking blocker, because this prevents getting useful crash stacks from testing the tryserver builds.

Comment 1

[(not currently active) Ted Mielczarek]

They probably don't have any debug symbols, so you're just getting export symbols, which are few and far between. Other libraries probably export more symbols, so they look more sane.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #1)
The release & nightly builds are the same aren't they ?

Comment 3

[(not currently active) Ted Mielczarek]

I would think so, yes, although maybe enabling debug symbols even though they're later stripped produces better results? Henri: can you try with a nightly and compare?

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Henri please respond to comment #3 so we can move on this.

Comment 5

[Henri Sivonen (:hsivonen)]

Sorry about the delay. I tried with a nightly. A nightly has a lot of missing symbols, but the symbols that do show up seem to be in the right places unlike with the try server builds.

Comment 6

[(not currently active) Ted Mielczarek]

I don't think there's anything different about the symbols. If they have the same amount of symbols, there's nothing I know of that would make them be placed at incorrect locations.

Comment 7

[Henri Sivonen (:hsivonen)]

Crash Reporter, Valgrind and Shark all show symbols in wrong places with try server builds. That's harmful compared to showing the absence of symbols, because getting lead to wrong trouble-shooting avenues due to bogus symbols is a time sink.

With nightlies, the symbols don't seem to appear in wrong places. I don't know how to explain why things happen this way.

If this could be fixed by making try server builds have full symbols, that would be awesome.

Comment 8

[Armen [:armenzg]]

I don't know how much this will help but here are the builds of building with the mozconfig that is used in the try server and the one used by the nightly builds.

https://build.mozilla.org/tryserver-builds/[e-mail]-try-mozconfig/
https://build.mozilla.org/tryserver-builds/[e-mail]-nightly-mozconfig/

Here are the logs respectively:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245878095.1245884378.15938.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1245874171.1245883012.13482.gz&fulltext=1

I will compare the logs tomorrow.

Is there a way for me to know if the builds have the symbols correctly in place? (I have never done this - if anyone want to help me with this it will be much appreciated)

Comment 9

[Armen [:armenzg]]

Attachment: diff and grep "no_symbols" from both logs

I have tried to compare the logs but it is difficult to know what to look for.

I am seeing some lines saying "no symbols" but it appears in both logs not just in one.

Comment 10

[Henri Sivonen (:hsivonen)]

It seems to me that both builds have bogus symbols. I profiled both builds in Shark while loading the HTML5 spec. The profiles showed that the builds spent most / second most time in "non-virtual thunk nsPrintSession::Release", which makes no sense. Also, the alleged callers seemed to be things that are unrelated to holding print sessions. (I didn't print anything.)

Comment 11

[Henri Sivonen (:hsivonen)]

This time both reported a sane root call tree around main, so I guess I should re-check the results with nightlies.

Comment 12

[Henri Sivonen (:hsivonen)]

The symbols in current nightlies are bogus in the same way. Chances are that my comment #5 was wrong, because the bogosity was/is less obvious than the even more obvious bogosity I had seen before. (Call stacks are rooted in the right kind of startup startup functions.)

Comment 13

[Benjamin Smedberg]

I think this bug report is INVALID. Neither the normal nightlies nor tryserver builds have debugging symbols included. For that you want the shark builds.

Comment 14

[Henri Sivonen (:hsivonen)]

The point of this bug is that it's harmful to get bogus stacks (stacks with symbols but wrong ones), because they cause time wasting. Stacks full of inexplicable numbers would not be actively harmful.

Comment 15

[(not currently active) Ted Mielczarek]

If it's the same as what we're shipping as nightlies, I think it's ok. See also bug 500007, which has requested full symbols for try server builds.