846864 - Source Server broken

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P2)

Description

[(not currently active) Ted Mielczarek]

The source server support in our Windows builds appears to be broken. I poked at the data in the pdb files, and the relevant bit (comparing Firefox 3.6 with a Nightly build) is:
-HGSERVER=http://hg.mozilla.org/releases/mozilla-1.9.2^M^M
+HGSERVER=e:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build^M^M

It looks like SRCSRV_ROOT isn't being set in the environment properly:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#549

Comment 1

[(not currently active) Ted Mielczarek]

Clarification: it seems like we may never have set SRCSRV_ROOT in the Mercurial world (that probably dates back to CVS), but we relied on `hg showconfig paths.default` pointing at the HTTP repo path.

Comment 2

[Chris AtLee [:catlee]]

`hg showconfig paths.default` does point to http://hg.m.o on the machines I've checked.

Note that you need to run it against the root of the checkout. If you run e.g. `hg -R /path/to/checkout/browser showconfig paths.default`, you get an empty result.

Comment 3

[(not currently active) Ted Mielczarek]

That should be ok, we're always passing in $(topsrcdir) here:
http://mxr.mozilla.org/mozilla-central/source/Makefile.in#141

I checked a few PDB files, it looks like this broke between Firefox 18 and 19. 18.0.2's PDB file looks fine:
$ /c/Program\ Files/Debugging\ Tools\ for\ Windows\ \(x64\)/srcsrv/pdbstr.exe
-r -p:firefox.18.0.2.pdb -s:srcsrv | grep HGSERVER
HGSERVER=http://hg.mozilla.org/releases/mozilla-release

but 19's is broken:
$ /c/Program\ Files/Debugging\ Tools\ for\ Windows\ \(x64\)/srcsrv/pdbstr.exe
-r -p:firefox.19.0.pdb -s:srcsrv | grep HGSERVER
HGSERVER=e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build

(and 19.0.1 as well).

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So what's the plan forward here?

Comment 5

[(not currently active) Ted Mielczarek]

It seems plausible that bug 774780 broke this. I'm not sure if backing that out is straightforward given other changes to symbolstore.py, but we can try that.

Comment 6

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I will send a bottle of the (not absurdly priced) drink of your choice to the person who fixes this bug.

Comment 7

[(not currently active) Ted Mielczarek]

Here's a crappy workaround. The caveat is that you have to run this on every symbol file you care about:
$ /c/Program\ Files/Debugging\ Tools\ for\ Windows\ \(x64\)/srcsrv/pdbstr.exe -r -p:firefox.pdb -s:srcsrv | sed -e 's/^HGSERVER=.*/HGSERVER=http:\/\/hg.mozilla.org\/releases\/mozilla-release/' > srcsrv

$ /c/Program\ Files/Debugging\ Tools\ for\ Windows\ \(x64\)/srcsrv/pdbstr.exe -w -p:firefox.pdb -s:srcsrv -i:srcsrv

(this also assumes that you're debugging release builds, you can swap whatever URL you need in there for HGSERVER.)

Comment 8

[(not currently active) Ted Mielczarek]

Attachment: Fix source server support

Not that I really expect you to know this code, but you care the most about this bug. :) I spent most of my time writing the test so I could sanity check my assumption.

Comment 9

[(not currently active) Ted Mielczarek]

Pushed this to try just because I was testing out bug 863715 anyway:
https://tbpl.mozilla.org/?tree=Try&rev=d3213be8d48b

Comment 10

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8d4bdcbc68c7

Comment 11

[(not currently active) Ted Mielczarek]

I sanity checked the symbols from my try server push and they look good.

Comment 12

[(not currently active) Ted Mielczarek]

I actually went all the way and loaded them in a debugger and was able to debug with source.

Comment 13

[(not currently active) Ted Mielczarek]

Comment on attachment 741472 [details] [diff] [review]
Fix source server support

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Regression from bug 774780.
User impact if declined: Makes debugging Windows crashes from crash-stats much more of a pain.
Testing completed (on m-c, etc.): Landed on mozilla-inbound, tested on a tryserver build.
Risk to taking this patch (and alternatives if risky): Very minimal. Change is only to the script that produces debug symbols. Patch includes tests.
String or IDL/UUID changes made by this patch: N/A

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8d4bdcbc68c7

Comment 15

[bhavana bajaj [:bajaj]]

Comment on attachment 741472 [details] [diff] [review]
Fix source server support

This patch looks good to land on aurora.

But given we will be going to build with our second last beta tomorrow & the fallouts from this *may* impact breakpad symbols somewhat in a wrong manner & since we heavily rely on symbols for crash-data, lets not take the risk on beta.

Also this is a FX19 regression.Would have been ok for early beta but seems late now to deal with the fallouts in our final beta's.

Please renominate if there is disagreement here.

Comment 16

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/releases/mozilla-aurora/rev/4ade17763d13