1181610 - Install debug symbol packages for system libraries on Linux testers

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[(not currently active) Ted Mielczarek]

bug 1173216 has an ASAN failure, and the stack is missing symbols from system libraries because we've never had symbols from system libraries. This doesn't help the stacks we get from crashes (because we'd need them in Breakpad format), but it will help ASAN, because that uses llvm-symbolizer which can deal with native debug symbols.

I did this once on a loaner slave by following the instructions here:
https://wiki.ubuntu.com/DebuggingProgramCrash#Non-built-in_debug_symbol_packages_.28.2A-dbgsym.29

It amounts to:
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
sudo apt-get update

Then it's just a matter of installing the right -dbg or -dbgsym packages. There's a script on the wiki for listing libs from a program using gdb, but having the list of filenames in the stack in the other bug I just ripped out the relevant bits and did:
(while read i; do dpkg -S $i; done | cut -f1 -d: | while read i; do for ext in dbgsym dbg; do apt-cache search "^$i-$ext\$"; done | head -1; done | sed -e 's/ - .*$//' | sort -u) <<EOF
/usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
/lib/x86_64-linux-gnu/libglib-2.0.so.0
/lib/x86_64-linux-gnu/libc.so.6
EOF

which produces:
libc6-dbgsym
libglib2.0-0-dbgsym
libgtk2.0-0-dbgsym

...which should be enough to get that bug unstuck.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

We have *tons* of gfx/gdk failures showing up as intermittents with long stack traces into gdk, cairo, libc, gobject, pixman, etc.  Often 25+ frames that are meaningless, and this makes it hard to even have an idea what's going wrong.  Random example: bug 1193570.  There are at least 10 or more in addition, and that's just ones I've seen in webrtc (they're happening elsewhere).


Please install debug symbols for *anything* firefox accesses, but especially the ones above.

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Randell Jesup [:jesup] from comment #1)
> We have *tons* of gfx/gdk failures showing up as intermittents with long
> stack traces into gdk, cairo, libc, gobject, pixman, etc.  Often 25+ frames
> that are meaningless, and this makes it hard to even have an idea what's
> going wrong.  Random example: bug 1193570.  There are at least 10 or more in
> addition, and that's just ones I've seen in webrtc (they're happening
> elsewhere).
> 
> 
> Please install debug symbols for *anything* firefox accesses, but especially
> the ones above.


This isn't going to fix those issues, sorry. This will only fix stacks for ASAN failures or assertion stacks that don't produce a crash. Stacks from minidumps are a different problem.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #0)
> libc6-dbgsym
> libglib2.0-0-dbgsym
> libgtk2.0-0-dbgsym

In the past, we've avoided installing extra libs on the testers to keep them as close as possible to what a typical user would be using. However, I think that's much less applicable on Linux.

cc-ing Rail since he's more familiar with the packaging on our Linux testers and how much work this would be.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Chris Cooper [:coop] from comment #3)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #0)
> > libc6-dbgsym
> > libglib2.0-0-dbgsym
> > libgtk2.0-0-dbgsym
> 
> In the past, we've avoided installing extra libs on the testers to keep them
> as close as possible to what a typical user would be using. However, I think
> that's much less applicable on Linux.

And doubly-so for debug symbol packages

> cc-ing Rail since he's more familiar with the packaging on our Linux testers
> and how much work this would be.

NI rail just to make sure it's noticed

Comment 5

[Rail Aliiev [:rail]]

This Should Not Be Hard (TM):

1) sync the ddeb repo to our mirror
2) add the repo to puppet, see http://hg.mozilla.org/build/puppet/file/b45ccae4dbf8/modules/packages/manifests/setup.pp#l192
3) create a meta class for all needed packages, something similar to http://hg.mozilla.org/build/puppet/file/b45ccae4dbf8/modules/packages/manifests/ia32libs.pp
4) include the class in modules/toplevel/manifests/slave/releng/test.pp 


Dustin, do we have enough disk space for step 1?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Unless you have a copy of the ddeb repo from the last time we sync'd the precise repos, I don't think that will work at all.

Probably better to just make sure this happens for the testing images we use in Taskcluster (/cc ahal).

If there are just a few ddebs that would be particuarly useful (e.g., the list in comment 0), and you can track down the right versions somewhere, then it may make sense to create a custom repo containing just those ddebs.

Comment 7

[(not currently active) Ted Mielczarek]

Attachment: debug-packages

I got a pair of loaners (Linux32, Linux64) and did some poking, and it turns out we have the -dbg packages in our apt mirror already, and they should be fine for our purposes. Attached is a list of -dbg packages for libraries that Firefox loads, I was able to install these on a loaner by just running:
xargs apt-get install -y < debug-packages

I haven't been able to get llvm-symbolize to produce the right results with these installed, though, so I'm not 100% confident it will fix the problem, but it shouldn't be hard to do.

Comment 8

[(not currently active) Ted Mielczarek]

rail: Given that these packages already exist in our mirror, can we just get them installed? I checked, it's the same list of packages for linux{32,64}.

Comment 9

[Rail Aliiev [:rail]]

It shouldn't be hard to get get installed. I have 2 questions before we can proceed:

1) do we need these on Talos slaves? If yes, would be great to let jmaher know.
2) will these impact the performance somehow? If yes, do we care?

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, cool!  Sorry to be such a downer then :)

In terms of the puppet config, I'd like to see these debug packages installed alongside the code that installs the base package, rather than a `pacakges::debug` that tries to install debug packages for "everything".  Otherwise we may find ourselves in a case where "everything" is different for different hosts.

Comment 11

[(not currently active) Ted Mielczarek]

(In reply to Rail Aliiev [:rail] from comment #9)
> 1) do we need these on Talos slaves? If yes, would be great to let jmaher
> know.

I don't think so, no.

> 2) will these impact the performance somehow? If yes, do we care?

There should be no impact other than using up some extra disk space (except in cases where we need to get symbols for an ASAN or assertion stack, where they will be loaded, but that's the point).