863461 - Socorro generates bad links in backtraces for files included from the build tree

Status: RESOLVED DUPLICATE of bug 671317

(Socorro :: Webapp, task)

Description

[Jim Blandy :jimb]

In crashes like this:
https://crash-stats.mozilla.com/report/index/ed3b5d55-cf5b-460e-b5b3-11caa2130418

the stack dump includes frames whose source links are dead. In that crash, the first stack frame links here:

http://hg.mozilla.org/releases/mozilla-aurora/annotate/4098c8763a3e/obj-firefox/x86_64/dist/include/js/Value.h#l872

This is taken directly from the stack dump produced by breakpad, but the link is bogus, because obj-firefox, the build tree, certainly doesn't appear in the Mercurial tree.

Some headers in the build tree are generated; clearly, Socorro can't really produce links to those. But others are (for reasons unknown to me) copied from the source tree into the build tree, so Socorro *could* generate good links to them. In particular:

obj-firefox/x86_64/dist/include/js      contains files from js/public
obj-firefox/x86_64/dist/include         contains files from js/src
obj-firefox/x86_64/dist/include/mozilla contains files from js/mfbt

So my ideal fix would remap those directories to links like:
http://hg.mozilla.org/releases/mozilla-aurora/annotate/4098c8763a3e/js/public/Value.h#l872
(which is a valid link), and not create links for anything else unrecognized in a build tree.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

Sorry, question carpet bombing:

so it is a simple mapping?  If we see "obj-firefox/x86_64/dist/include/js" we replace it with "js/public" when we generate the link?  

Are there any exceptions to the mapping rules?  Are we likely to see generated headers in these directories for which we cannot/should not make links?  If so, how do we tell them apart from the "copied" header files? 

Is this list of mappings likely to expand to more than three entries? 

> not create links for anything else unrecognized in a build tree.

Does the "build tree" always begin with "obj-firefox"?  How do we determine what is "unrecognized"?  What about the other Mozilla products, do they have the same/similar rules?

Comment 2

[Jim Blandy :jimb]

I think it's a simple mapping. Here's from js/src/Makefile.in:

######################################################
# BEGIN SpiderMonkey header installation
#
# Mozilla/Gecko/Firefox mostly doesn't concern itself with defining a sensible
# install target, because it's shipping primarily packaged builds.  And even if
# it did, those builds wouldn't really have reason to include header files.  So
# we have to install public headers ourselves, rather than using something from
# config/rules.mk or similar.
#
# The overall directory structure of the installed headers looks like so:
#
#   $(includedir)/
#      $(MODULE)/
#        jsapi.h, jspubtd.h, etc. (all of INSTALLED_HEADERS)
#        js/
#          js/public/* headers (all are public)
#        ds/
#          js/src/ds/* public headers
#        gc/
#          js/src/gc/* public headers
#        mozilla/
#          mfbt headers
#
# Currently $(MODULE) is just js, but we'll soon be changing it to include the
# SpiderMonkey version, so that headers for multiple SpiderMonkey installs can
# exist on a system at once.  (This permits embedders depending on system
# SpiderMonkey to independently update their embeddings to newer SpiderMonkey
# releases at different paces.)
#
# It is expected that embedders will then add -I$(includedir)/$(MODULE) to
# their compiler invocations, which will then expose all of jsapi.h and friends
# as they've always been exported.  (This will happen naturally for anyone using
# pkgconfig systems to determine compiler/linker flags and the like.)  Headers
# in subdirectories will show up as "mozilla/Assertions.h" and so on.  Also,
# this system keeps #include "js/CharacterEncoding.h" in internal headers
# working -- not that this is required, but it does simplify things.  (At some
# point we may want to convert SpiderMonkey, and maybe Gecko, to this system, in
# the interest of being a more conventional embedder, but it's probably a ways
# off.)
#
# Eventually we'll probably trim this structure down to having the mfbt headers
# in mozilla/, and every SpiderMonkey-specific header in js/*.  In the meantime
# this is the structure implemented below.

> Are there any exceptions to the mapping rules?

From the above, I think not.

> Are we likely to see
> generated headers in these directories for which we cannot/should not make
> links?

There are some, but they're mostly copied files, so I think it would be an overall improvement to do the remapping, even knowing that it will catch some generated files. If it's feasible (does Socorro have an hg working copy at hand?), Socorro could just *check* if the remapped file actually exists at that changeset.

> If so, how do we tell them apart from the "copied" header files? 

Unless we actually check against an hg checkout, I don't think there's a good way. But doing the mapping in a blanket way will reduce the absolute number of dead links, so while it's not correct, it's "correcter" than we are now.

> Is this list of mappings likely to expand to more than three entries? 

From the comment, it looks like there are others.

> > not create links for anything else unrecognized in a build tree.
> 
> Does the "build tree" always begin with "obj-firefox"?  How do we determine
> what is "unrecognized"?  What about the other Mozilla products, do they have
> the same/similar rules?

I have no idea; it depends on how the configure script is run, so it's not even something determined by the in-tree build files.

I think what we're looking for here is not correctness, but an improvement over the status quo. It's certainly true that not having reliable information on the mapping means that we'll have to revisit the mapping Socorro uses as the source tree evolves (adds new copied directories, perhaps), and that's an ongoing cost to fixing the dead links that should be acknowledged.

But as I say, if we're willing to take the hit of doing this remapping (I don't know, but I'm hoping it's just a localized change), then we can have fewer dead links than we do now.

Comment 3

[Robert Kaiser]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #1)
> so it is a simple mapping?  If we see "obj-firefox/x86_64/dist/include/js"
> we replace it with "js/public" when we generate the link?  

That might be the optimal way to do it, but if we want to be less fragile, we just refrain from linking files that are in objdirs.
(I also consider this straight copying from source to objdir that they are doing to be pretty bad practice anyhow, but that's written on a different page.)


> Does the "build tree" always begin with "obj-firefox"?

Looks like it starts with "obj-firefox" for Firefox, with "objdir" for SeaMonkey, with "objdir-tb" for Thunderbird, "obj-b2g" for B2G.
The good thing is that we can pretty safely assume that no source directory actually starts with "obj-" or "objdir", at least in the Mozilla trees.

> How do we determine what is "unrecognized"?

Ideally, we'd look at the connected source tree and see if a file is there. But that's too much here, I'm pretty sure. I think we just shouldn't link anything starting with "obj-" or "objdir" (maybe 'and containing "/dist/"'), just like we already omit linking in the cases of full path like the one in bp-41669b38-1333-421a-b2b3-f9d3b2130419.

> What about the other Mozilla products, do they have the same/similar rules?

For Mozilla products, that's the same in all of them, AFAIK.

Comment 4

[Jim Blandy :jimb]

(In reply to Robert Kaiser (:[e-mail]) from comment #3)
> That might be the optimal way to do it, but if we want to be less fragile,
> we just refrain from linking files that are in objdirs.
> (I also consider this straight copying from source to objdir that they are
> doing to be pretty bad practice anyhow, but that's written on a different
> page.)

That would maximally reduce bad links, and it's low-maintenance, so I couldn't blame you for going that route. But a little remapping would improve your good link count, and sure would be nice.

Comment 5

[Peter Bengtsson [:peterbe]]

This bug is over 3 years old. Is this still a think that happens?

I think it shouldn't be that hard to maintain a list of (about 4) string replacements to make on the links. 

Having an hg checkout does not feel like a small feat at this point. If the string replacements are well documented and unit tested it would be an easy thing to add to the processor rules.

Comment 6

[Jim Blandy :jimb]

Yes, here's an example:

https://crash-stats.mozilla.com/report/index/b8d68d01-b17b-424f-b1fa-fcbf32160525

Click on the source link for frame 0:

https://hg.mozilla.org/mozilla-central/annotate/829d3be6ba64/obj-firefox/dist/include/js/GCHashTable.h#l273

I get an hgweb page that says:

An error occurred while processing your request

obj-firefox/dist/include/js/GCHashTable.h@829d3be6ba64: not found in manifest

Comment 7

[(not currently active) Ted Mielczarek]

Longstanding issue, unfortunately. We should be smarter and at least not generate vcs info for files in the objdir. That should be fairly easy to fix. Actually making these links go to the source is harder because we don't currently upload the generated sources anywhere.