1460306 - Please also check symbols.mozilla.org/try/... for symbol files

Status: VERIFIED FIXED

(Socorro :: Symbols, task)

Description

[Markus Stange [:mstange]]

Bug 1437577 just landed and gave developers a way to upload symbols for tryserver builds to the symbol server, see the announcement email: https://groups.google.com/d/msg/mozilla.dev.platform/Ob1UGcSCH8A/9jA98n2EAgAJ

It would be nice to have these symbols available in tecken. They are hosted at a different URL (inside the "try" directory) and have a different retention policy.

Here's an example request for an address in mozglue.pdb from the try build at https://treeherder.mozilla.org/#/jobs?repo=try&revision=a26b4bfc39888a8d62af15e56ae5da81c177198b

> curl 'https://symbols.mozilla.org/symbolicate/v5' --data '{"memoryMap":[["mozglue.pdb","1BA997233B404BFDBDBABF25FA56CD9D1"]],"stacks":[[[0,192696]]]}'

It currently returns:

{"results":[{"stacks":[[{"module_offset":"0x2f0b8","module":"mozglue.pdb","frame":0}]],"found_modules":{"mozglue.pdb\/1BA997233B404BFDBDBABF25FA56CD9D1":false}}]}

Comment 1

[Peter Bengtsson [:peterbe]]

Any preferences or suggestions for the best way to tell the symbolication server that you want to draw from Try symbols too? 

Here's how it's done for regular symbol download http://tecken.readthedocs.io/en/latest/download.html#try-builds

We could do:

> curl 'https://symbols.mozilla.org/symbolicate/v5/try' --data '{"memoryMap":["mozglue.pdb","1BA997233B404BFDBDBABF25FA56CD9D1"]],"stacks":[[[0,192696]]]}'

or 

> curl 'https://symbols.mozilla.org/symbolicate/v5' --data '{
"try": true, "memoryMap":["mozglue.pdb","1BA997233B404BFDBDBABF25FA56CD9D1"]],"stacks":[[[0,192696]]]}'

Comment 2

[Markus Stange [:mstange]]

I would prefer not to have to tell the symbolication server that I want it to draw from Try symbols, too, because I always want it to. At the moment there's no simple way for perf.html to detect that it's looking at a profile from a Try build, so always checking for Try symbols would be the easiest solution for me.

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Markus Stange [:mstange] from comment #2)
> I would prefer not to have to tell the symbolication server that I want it
> to draw from Try symbols, too, because I always want it to. At the moment
> there's no simple way for perf.html to detect that it's looking at a profile
> from a Try build, so always checking for Try symbols would be the easiest
> solution for me.

Interesting. I just wonder if it's easy to remember that rule. If the Download part of Symbols requires explicit things to draw from Try symbols, it might be confusing that the Symbolication part does it entirely differently. 

The reason we made Try symbols not on by default in Download is because of security. A rogue symbol that gets into symbols.mozilla.org by a rogue user, might end up on an unsuspecting users debugger and cause harm. At least that's what I remember the logic being. 

How about this; just like you already have the whole URL hardcoded, you just change it from https://symbols.mozilla.org/symbolicate/v5 to https://symbols.mozilla.org/symbolicate/v5/try

Comment 4

[Markus Stange [:mstange]]

> might end up on an unsuspecting users debugger and cause harm

This is luckily not a problem for us. Ted earlier wrote the following on IRC:

> 10:41 <ted> making the symbolicate api use try symbols should always be OK
> 10:42 <ted> we split it out for the symbol server part because microsoft's debuggers can execute arbitrary commands from pdb files :-/

The commands that we're worried about are in the pdb files, not in the sym files. Tecken only parses the sym files, which are plain text.

Comment 5

[(not currently active) Ted Mielczarek]

Right, the separation is only important for downloading actual PDB files in a Microsoft debugger. For any other purpose it should be fine for Tecken to return results from the try bucket as well. Assuming Tecken looks in the standard bucket first (so that symbols from try builds won't intentionally or accidentally override symbols from published builds) there should be no issues.

Comment 6

[Peter Bengtsson [:peterbe]]

Cool. Then let's just do it. The Symbolication will do what the Download does with ?try enabling.
Always Try symbols. But, regular symbols first if there's a name clash. Just like the Download.

Comment 7

[(not currently active) Ted Mielczarek]

I think it'd be nicer if the /symbolicate API always considered Try symbols. Would it be straightforward to do that?

Comment 8

[Peter Bengtsson [:peterbe]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7)
> I think it'd be nicer if the /symbolicate API always considered Try symbols.
> Would it be straightforward to do that?

Yes. I'll leverage the code from Download but reverse the default of it being in use. Or rather, just hardcode it on always (...in the context of Symbolication).

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/876

Comment 10

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/80e4c7e3e25dc7fde1b15069093ea87a91d45500
Fixes bug 1460306 use try symbols in symbolication (#876)

* fix filter function for Uploads frontend

* fixes bug 1460306 - use Try symbols in symbolication

Comment 11

[Peter Bengtsson [:peterbe]]

I appreciate that there's no way to test this since there are no Try symbols other than in Prod. 

So let's make a release. 

However, it's Friday noon and there are a couple library updates in the pipeline. So I'll try to get this into Prod next week.

Comment 12

[Peter Bengtsson [:peterbe]]

The good news is that this appears to work on Stage. 
I grabbed the latest .zip file from prod that as a Try upload. Then I uploaded that to Stage and now this works:

curl -XPOST -d '{"jobs":[{"memoryMap":[["AccessibleHandler.pdb","5A3477C58D6144A388B3F618C5ADDD452"]],"stacks":[[[0,192696],[0,10000]]]}]}' https://symbols.stage.mozaws.net/symbolicate/v5

It didn't find it after I tried the first time and now it finds it. (nice that the caches were immediately updated after the upload)

By the way, There's a new frontend UI tool! Go to https://symbols.stage.mozaws.net/ and click Symbolication. It allows you to basically do what you do in curl but you do it via a really simple web form. It's probably buggy and not particularly useful but it can really support the documentation I hope.

Comment 13

[Peter Bengtsson [:peterbe]]

Markus, 
Can you please verify?

Comment 14

[Markus Stange [:mstange]]

I can confirm it's working. Thanks!

I reproduced it with the "OS X Cross Compiled opt" build from https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c389b844507b73bcf38ef5f07ff894bc74d540c . I downloaded the build and got a profile and saw "XUL/0C4DBA8CF3C13FB39DB0993DA5FBBBB50": false in the foundModules map of the symbolication API network response. Then I triggered the Sym job, waited for a few minutes, got a new profile, and saw "XUL/0C4DBA8CF3C13FB39DB0993DA5FBBBB50": true in the foundModules map of the symbolication API network response, and the symbols were correct.

Comment 15

[Peter Bengtsson [:peterbe]]

That makes me very happy. Thanks for verifying.