1518254 - Symbolication no longer works on Windows

Status: RESOLVED FIXED

(Tecken :: General, task)

Description

[Jeff Muizelaar [:jrmuizel]]

I noticed this on two different machines today. The profiles just get xul.dll and addresses: https://perfht.ml/2VBiUzi

Comment 1

[Jeff Muizelaar [:jrmuizel]]

I'm getting a bunch of "The symbolication API request was not successful for xul.pdb/3F74EC5B36CBB0170D6584A234CD13A31: SymbolsNotFoundError: "There was a problem with the JSON returned by the symbolication API." type errors.

Comment 2

[Greg Tatum [:gregtatum]]

Minimal STR:

Using curl from with a Mac library:

curl 'https://symbols.mozilla.org/symbolicate/v5' \
--data '{"memoryMap":[["libGLImage.dylib","B5464F1AB9363BEE97CF3806AD1A9E680"]],"stacks":[[[0,27278]]]}'

Response:

{"results":[{"stacks":[[{"module_offset":"0x6a8e","module":"libGLImage.dylib","frame":0}]],"found_modules":{"libGLImage.dylib\/B5464F1AB9363BEE97CF3806AD1A9E680":false}}]}

Using curl with a Windows library:

curl 'https://symbols.mozilla.org/symbolicate/v5' \
--data '{"memoryMap":[["xul.pdb","7F0A9D9E05F2045B2ACDE186459DCDFC1"]],"stacks":[[[0,20825192]]]}'

Response:

{"error": "Internal Server Error"}

Comment 3

[Greg Tatum [:gregtatum]]

Hey Peter, this looks like something that's going on with symbols.mozilla.org. Do you have any ideas?

Comment 4

[Peter Bengtsson [:peterbe]]

(In reply to Greg Tatum [:gregtatum] from comment #3)

Hey Peter, this looks like something that's going on with symbols.mozilla.org. Do you have any ideas?

I made a release today. It contained various rather uninteresting python (and some frontend) upgrades.

The error is: ValueError: 204599 exceeds max_array_len(131072)
https://sentry.prod.mozaws.net/operations/symbols-prod/issues/5100624/

I'll try to figure it out. Tomorrow.

Comment 5

[Peter Bengtsson [:peterbe]]

I can reproduce it locally. The problem is that msgpack.unpackb() can't turn this stream of bytes into a list of integers.

I and override this by using msgpack.loads(stream, max_array_len=2**32 - 1).

If I do that I discover that there are 204,599 valid keys in the xul.pdb/7F0A9D9E05F2045B2ACDE186459DCDFC1/xul.sym file which I suspect isn't the first time but the first time we've tried to use extract it from the cache.

Opting for simplest and most brutal solution for now.

Comment 6

[Peter Bengtsson [:peterbe]]

Fixed with https://github.com/mozilla-services/tecken/pull/1463 but I'll close this issue once that goes out to Prod.

Comment 7

[Peter Bengtsson [:peterbe]]

Attachment: Screen Shot 2019-01-08 at 1.29.30 PM.png

This was not working before. Now it does.

Comment 8

[Greg Tatum [:gregtatum]]

Thanks so much Peter for the help!

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving from Socorro product to Tecken product.