Open Bug 1088809 Opened 10 years ago Updated 9 years ago

Source server for gdb

Categories

(Developer Services :: General, task)

Product:
Developer Services
Component:
General
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

People

(Reporter: sfink, Unassigned)

References

(Depends on 1 open bug)

Details

Here's what I want to be able to do:

1. Go to crashstats, see a crash that I want to look at. Find what build it's for. Download and install the build on my Linux box.
2. Struggle with fetch-symbols.py for a while to download the corresponding symbols. (Yes, I should file another bug for the symbol server part of this.)
3. Start up the browser under gdb. Hit a crash.
4. Smile as gdb automagically lazily fetches the source for any frame into a temporary area (named by the revision number so it'll be cached across restarts) and displays it.

Ok, honestly I would be running gdb inside of emacs while doing this, so step 4 would just happen instead of requiring me to type 'list' or something.

I am wondering how far off this is with the current gdb/python integration.

I'm not sure the best way for gdb to figure out what version of source to fetch, but the binary contains it somewhere. (about:buildconfig contains the link.) If nothing else, we can add a .ssurl section or something.
There's no way in gdb to use Python to intercept the source lookup step.
So a purely gdb solution can't readily be done.

However, the necessary infrastructure is all there if you use build-ids and
some network file system (personally I'd look at some fuse-based thing since
they're easier on the user end; but anything would do).

For convenience some build-id docs are here
http://fedoraproject.org/wiki/Releases/FeatureBuildId
.. but there's also stuff in the gdb manual and elsewhere.
Fedora has been using this for years now, though Fedora's needs aren't
quite identical to Mozilla's; and in particular Fedora is more
download-oriented (there was a stab at fixing this but never deployed,
see "darkserver") and has some lingering issue so that debuginfo can
only be installed for a single revision of a package.


Anyway, I think a promising route would be:

* Enable build-ids in Linux builds.  Easy; the default on Fedora,
  not sure about other distros.  You can verify with:

readelf -n ./obj-x86_64-unknown-linux-gnu/dist/bin/firefox

  and see something like
  
Notes at offset 0x0000023c with length 0x00000024:
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 46fb249fc525c52afac6a584b424a7845e6958a2

  (I didn't look to see whether Mozilla is already doing this.)

* When doing a build, use -fdebug-prefix-map to give the compilation
  directory a name that is based on the source revision.  We'll use this
  later to fetch the sources.

* When stripping the debuginfo, set up a tree in the filesystem following
  the build-id rules.  https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

* Export the debuginfo tree verbatim.

* Arrange it so that requests to the rewritten compilation tree look in mercurial
  to extract the correct version of the source.


To use this server, a user would mount the remote filesystem.
Then in gdb, "set debug-file-directory /path/to/remote/fs".
After this gdb will find the needed debuginfo using the build-id.
The debuginfo will encode the correct version of the sources in the
compilation directory names.


Ok, well, I think it would all work :)

I am not sure whether relative directories would work ok for -fdebug-prefix-map.
If not then I think this would require all users to use the same absolute
path for their mount, which would be a bummer.

It would be best if the filesystem did some local caching.  All the data can
be readonly.

With this setup, configuration is pretty minimal and even the "download a build"
step can be skipped.  You can just run it straight from the remote filesystem.


I think it would be possible to approach this incrementally.  E.g., start with build-ids
and keep a map from build-ids to source revisions.  Then go from there.

Another possible twist is to use a local fuse-over-hg filesystem (no idea if such exists)
rather than querying the build-id server for sources.
One other thing that occurred to me is that builds intended for release & distribution
(as opposed to one-off developer builds) would probably benefit from enabling the
gdb-index feature.  This adds to the debug info size but doesn't affect the stripped
binaries; but in return it usually speeds up gdb quite a bit.

Some info on this here:
https://sourceware.org/gdb/onlinedocs/gdb/Index-Files.html
(In reply to Tom Tromey :tromey from comment #1)
> readelf -n ./obj-x86_64-unknown-linux-gnu/dist/bin/firefox
> 
>   and see something like
>   
> Notes at offset 0x0000023c with length 0x00000024:
>   Owner                 Data size	Description
>   GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
>     Build ID: 46fb249fc525c52afac6a584b424a7845e6958a2
> 
>   (I didn't look to see whether Mozilla is already doing this.)

Yes, Mozilla has been using build ids for a while now.

> Ok, well, I think it would all work :)

It all sounds vaguely sane to me. Though it's too bad we can't embed http: or git: urls and have gdb call python when resolving paths with schemes in them.

> I think it would be possible to approach this incrementally.  E.g., start
> with build-ids
> and keep a map from build-ids to source revisions.  Then go from there.

I have dim memories of someone requesting this map. Or saying that it exists. Or once did, but was unmaintained.

> Another possible twist is to use a local fuse-over-hg filesystem (no idea if
> such exists)
> rather than querying the build-id server for sources.

There's an ancient-looking one at https://code.google.com/p/hgfs/ though I suspect its main purpose was for write access. There's gitfs, which we could use with our git mirror.
Just FYI, the source server info on Windows is generated specially at build time:
http://hg.mozilla.org/mozilla-central/annotate/20408ad61ce5/toolkit/crashreporter/tools/symbolstore.py#l649

http://hg.mozilla.org/mozilla-central/annotate/20408ad61ce5/toolkit/crashreporter/tools/symbolstore.py#l649

We stick a stream in the PDB that contains the list of all source files it references, and a way to map those to a URL to fetch the matching source.

Presumably we could invent something similar to stick in the debug info on GDB-using platforms that we could use to locate source.
See also the Fedora "darkserver" project.  It's a web-based server
that provides build-id to debug-data lookup/download services.
http://fedoraproject.org/wiki/Darkserver
(In reply to Steve Fink [:sfink, :s:] from comment #0)
> 2. Struggle with fetch-symbols.py for a while to download the corresponding
> symbols. (Yes, I should file another bug for the symbol server part of this.)

This part at least is easy now with gdb 7.9:
https://gist.github.com/luser/193572147c401c8a965c

You can source that script in gdb 7.9 and debug a nightly/release and it will fetch the symbols for you automatically. Hooray!

I'd still love to get the source server bit working.
So tromey pointed me at the right place, and I have a gdb patch that makes this possible:
https://github.com/luser/binutils-gdb/commit/872f947c49a13481c781f23b1efcdefaaa81ab0c

I'm going to submit it upstream once I get tests and docs written. It adds a "gdb.find_source_hook" to the Python API which will be called if gdb can't locate a source file. If you return a filename gdb will use that as the source file.

tromey also pointed me to the madness that is .debug_gdb_scripts: https://sourceware.org/gdb/current/onlinedocs/gdb/dotdebug_005fgdb_005fscripts-section.html

so I wrote this little script, which can take a list of source file->http URL mappings and write them into an ELF section that can use that hook to magically retrieve source files via HTTP:
https://gist.github.com/luser/d09bdee635872d1cc851

If we distributed the script from comment 6 alongside the firefox binary, and merged the script here into our symbolstore.py script that does the objcopy stuff to split out debug symbols, this would totally work (once everyone has a GDB with my patch in it).
(It would be handy to have a hook from gdb to allow scripts to search for even that bit of initial debuginfo that identifies the source file names.)
There are probably nicer things we could do here, certainly, but making this work with Python extension scripts is a good first cut.
You need to log in before you can comment on or make changes to this bug.