1389225 - Support fetching generated source files from S3 and Rust standard library sources in source server indexing

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[(not currently active) Ted Mielczarek]

In bug 1259832 I'm making Firefox builds upload generated source files to S3 so we can link to them from crash reports. We should also be able to make Microsoft debuggers fetch them using the source server indexing we add to our PDB files. However, in my patches there I punted on actually doing that because the existing code looks a little fragile:
https://dxr.mozilla.org/mozilla-central/rev/4c5fbf49376351679dcc49f4cff26c3c2e055ccc/toolkit/crashreporter/tools/symbolstore.py#342

Microsoft actually has some MSDN documentation on the source server indexing bits nowadays:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/http-sites-and-unc-shares
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/language-specification-1

The quirk is that we'd have to be able to return one of two different URLs depending on whether the file is in hg or in s3. I think this is possible to do with the way SrcSrv does variable evaluation, but it might be a little fiddly.

Comment 1

[(not currently active) Ted Mielczarek]

I realized while debugging a minidump today that we don't have source server info for sources from the Rust standard library either, because we explicitly only include files in hg:
https://dxr.mozilla.org/mozilla-central/rev/0cd106a2eb78aa04fd481785257e6f4f9b94707b/toolkit/crashreporter/tools/symbolstore.py#571

We have info in symbolstore.py that we use to map Rust sources to the proper revision on GitHub, but we only put that info in the .sym files:
https://dxr.mozilla.org/mozilla-central/rev/0cd106a2eb78aa04fd481785257e6f4f9b94707b/toolkit/crashreporter/tools/symbolstore.py#462

Fixing this is probably the same amount of work as fixing the generated sources bits.

Comment 2

[Markus Stange [:mstange]]

Ted created a demo for how these different paths could be expressed in the srcsrv stream:

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
SRCSRV: variables ------------------------------------------
HGSERVER=https://hg.mozilla.org/mozilla-central
HG_REV=1706d4d54ec68fae1280305b70a02cb24c16ff68
SRCSRVVERCTRL=http
HG_EXTRACT_TARGET=%hgserver%/raw-file/%hg_rev%/%var3%
S3_TARGET=https://gecko-generated-sources.s3.amazonaws.com/%var3%
RUST_GITHUB_TARGET=https://github.com/rust-lang/rust/raw/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/%var3%
SRCSRVTRG=%fnvar%(%var2%)
SRCSRV: source files ---------------------------------------
/builds/worker/checkouts/gecko/mozglue/baseprofiler/core/ProfilerBacktrace.cpp*HG_EXTRACT_TARGET*mozglue/baseprofiler/core/ProfilerBacktrace.cpp
/builds/worker/workspace/build/src/obj-firefox/dom/bindings/AddonManagerBinding.cpp*S3_TARGET*d89a77356015cf19cbe2488905a8e878d105c1d2b8beaf233409a976cce65a065b8319b87b7bb6e2a1dd4c5b7c0c594b9e2c171c47b6dbfe17ec0b08dc453730/dom/bindings/AddonManagerBinding.cpp
/rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs*RUST_GITHUB_TARGET*src/liballoc/boxed.rs
SRCSRV: end ------------------------------------------------

Comment 3

[Markus Stange [:mstange]]

Attachment: Bug 1389225 - Extend the PDB srcsrv info to cover S3 and rust github paths. r=gsvelto

This will make the source code show up in Visual Studio when
debugging using the Mozilla symbol server.

In the past, only source files from hg.mozilla.org were displayed,
but no generated sources and no Rust standard library sources.

This change adds the required annotations for those two missing cases.

The generated sources will be pulled from S3.
The Rust standard library sources will be pulled from github.

Depends on D154299

Comment 4

[Markus Stange [:mstange]]

https://treeherder.mozilla.org/jobs?repo=try&revision=acbc54e0914c3951f517ba4f9a738bb9957fcad3

Comment 5

[Markus Stange [:mstange]]

target.zip from the Windows try build
This build should have full sources in the debugger, and full sources in a profile from etw-gecko.

Comment 6

[Markus Stange [:mstange]]

I ran into bug 1784324 while testing this on Try. I'll re-test once this is on mozilla-central.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/336eb0c88c68
Extend the PDB srcsrv info to cover S3 and rust github paths. r=gsvelto

Comment 8

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #5)

target.zip from the Windows try build
This build should have [..] full sources in a profile from etw-gecko.

etw-gecko profile: https://share.firefox.dev/3SICwPo

I think it would work, if it weren't for bug 1784324 and bug 1784340.

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/336eb0c88c68