1405713 - Can't load files from srcsrv

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(Away)]

0:040> !sym noisy
noisy mode - symbol prompts on
SRCSRV:  z:\build\build\src\js\src\jsapi.h not indexed

Comment 1

[(Away)]

13:19:52 <•ted> https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#558
13:19:55 <•ted> i bet that line is to blame
13:20:04 <•ted> needs to be swapped with the `sourcepath = filename` line below

Comment 2

[(Away)]

Attachment: srcsrv

Do these look okay? https://treeherder.mozilla.org/#/jobs?repo=try&revision=4b468f0efda629c769baf2901889522fce05f75b

unit-symbolstore.py passed on win32, win64, linux64. It appears not to have run on the mac build.

Comment 3

[(not currently active) Ted Mielczarek]

(In reply to David Major [:dmajor] from comment #2)
> Do these look okay?
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=4b468f0efda629c769baf2901889522fce05f75b

I grabbed firefox.pdb from a recently nightly for a baseline. I ran pdbstr to extract the srcsrv stream:
'/c/Program Files (x86)/Windows Kits/10/Debuggers/x64/srcsrv/pdbstr.exe' -r -p:firefox.pdb -i:srcsrv.stream -s:srcsrv

$ grep -i nsbrowserapp srcsrv.stream
Z:\task_1507497040\build\src\browser\app\nsBrowserApp.cpp*browser/app/nsBrowserApp.cpp*5eba13f5b3a6

The basis of this bug is that the debugger is looking for the exact paths that show up in the PDB's source file list, and those  paths start with "z:\build":
$ /c/build/breakpad/src/src/tools/windows/binaries/dump_syms.exe ./firefox.pdb 2>/dev/null | grep -i nsBrowserApp
FILE 454 z:\build\build\src\browser\app\nsbrowserapp.cpp

(We build Taskcluster Windows builds in z:\build which is a symlink so the z:\task_nnn directory.)

I downloaded the symbols-full.zip from the Win32 build on your try push:
https://queue.taskcluster.net/v1/task/P03VVW0RRAmYWhawbfDf5g/runs/0/artifacts/public/build/target.crashreporter-symbols-full.zip

Running those same commands on the firefox.pdb contained within (after running it through `expand`) shows that the paths in the srcsrv stream are correct:
$ grep -i nsbrowserapp srcsrv.stream  ows/binaries/dump_sym
z:\build\build\src\browser\app\nsbrowserapp.cpp*browser/app/nsBrowserApp.cpp*4b468f0efda6

ted@DESKTOP-32V9ND8 /tmp/firefox.pdb/C54268FB1C3841EF83DF3BFF834127051
$ /c/build/breakpad/src/src/tools/windows/binaries/dump_syms.exe ./firefox.pdb 2>/dev/null | grep -i nsBrowserApp
FILE 24 z:\build\build\src\browser\app\nsbrowserapp.cpp

Spot-checking firefox.sym shows that header files from dist/include still have VCS info added, like:
FILE 15 hg:hg.mozilla.org/try:mfbt/Vector.h:4b468f0efda6

And checking xul.sym shows that generated source files are still being pointed at the s3 bucket:FILE 468 s3:gecko-generated-sources-l1:0f2e1cbae30f0e494e0969feae9b3080212fa60dd3b7e214e3e3bc6bc7501f1fa5a9e80f7c2761338
cb5203efff9e2840393aba5ad3a30ac6796bcedc01849a4/dist/include/nsIEventTarget.h:

> unit-symbolstore.py passed on win32, win64, linux64. It appears not to have
> run on the mac build.

We don't run any tests on the mac builds because they're cross-compiles. We probably should fix things so we can run some things, the integration test in unit-symbolstore.py would still be useful to run.

Comment 4

[(Away)]

Thanks for the detailed explanation, Ted!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4b468f0efda629c769baf2901889522fce05f75b

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/06f38c2c91c4
Reorder sourcepath normalization to fix srcsrv. r=ted

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/06f38c2c91c4

Comment 7

[(not currently active) Ted Mielczarek]

[Tracking Requested - why for this release]:

This was a regression from bug 1392312. This is a simple one-line fix that only affects the content of the debug symbols we upload to the symbol server (not the symbols we use for crash processing). However, it makes debugging release builds and minidumps from release builds much easier on Windows. I think we should uplift it for 57.

Comment 8

[(Away)]

Comment on attachment 8917030 [details] [diff] [review]
srcsrv

Approval Request Comment
[Feature/Bug causing the regression]: Bug 1392312
[User impact if declined]: Engineers can't see source files when debugging Firefox
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: Yes; I confirmed the broken behavior in the 2017-10-11-AM build and that it's fixed in the 2017-10-11-PM build.
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: No
[Is the change risky?]: No
[Why is the change risky/not risky?]: See comment 7
[String changes made/needed]: No

Comment 9

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8917030 [details] [diff] [review]
srcsrv

Definitely a great idea to uplift this, beta57+

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f66f3fd67100