1429516 - Store each symbol offset/signature in the Redis store

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

Our current Symbolication service essentially gets a request for symbols + offsets. 
What Tecken does is that it downloads the symbols from S3, parses those plain text files and generates a python dict of offsets (keys) signatures (values). 
Since the download is network IO bound and slow, the Redis store is the cache where we store the download for the next time it's needed. How it does it is that it makes the symbol (e.g. xul.pdb/366A8BA696B5451BA9148B24B61FD3B92) the key and the python dict of all offsets and signatures the value (e.g. {"10001": "C++ function1", ...}).

After some testing of this, we found it's quite slow. I.e. about 300-400ms per request (within Django). The primary reason it's slow (at least when load testing with sample crash ping stacks) is that we often have to rely on some MASSIVE symbols (e.g. xul.pbd/HEX), even though we only need a handful of offsets from within it. For example, a xul.pdb file, when parsed, is likely to contain about 220K offsets. 

Side-note: A sample xul.pdb symbol file...
* raw: 113MB
* turned into a dict and dumped with Msgpack as a string: 19.8MB
* ...and zlib compressed: 3.6MB

Side-note II: The reason the initial caching design was to store and retrieve the whole python dict was that we need access to ALL offsets per symbol. If the offset we're looking for can't exactly be found, we need the nearest offset (the one before). 

So if we have to retrieve the whole 3.6MB of data from the Redis store, but only use a couple of its key-values, we're significantly wasting network IO. 

An alternative solution is to use hash maps. This is based on the assumption that we need significantly fewer keys from within each symbol file than the file has.

One hash map per symbol. Within, every offset becomes a key and the signatures the individual values. 
We still need to know all the possible keys for when we need to look up the nearest offset instead. 

Side-note III: If msgpack'ing and zlib compressing the whole python dict is about 3.6MB. Doing the same with just the list of all keys is 560KB.

Comment 1

[Peter Bengtsson [:peterbe]]

Measuring this is hard. The only environment that has real symbols is Prod. But Prod is not a good place to do load testing. 
Locally, I do have access to our production S3 bucket but on a home broadband the S3 download is sloooow. 
Also, when doing development locally, the Redis server is on "localhost" so the latency is very low. 

I ran a bunch of symbolication for a while and gathered some numbers::

https://gist.github.com/peterbe/3c2055ada6680e8b8c67aed2ed7254d0

The conclusion is that local symbolication spends roughly on average 0.117s making Redis store queries to get the symbols.

Comment 2

[Peter Bengtsson [:peterbe]]

The results aren't super promising but it's hard to measure this.
I ran 5 symbolications with the old way and 5 with the new way. All things downloaded so the total times aren't affected by network IO to the symbols. 

On average, the new method takes 59.6ms. The old method 134.1ms per complete lookup.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/688

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/d55e01e97e7b8734eb672bcc0c10f34581012671
fixes bug 1429516 - cache individual symbol map keys (#688)