1228122 - Optimize and abstract graphics devices lookup

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Peter Bengtsson [:peterbe]]

Much of the plan laid out here: https://github.com/mozilla/socorro/pull/3113#issuecomment-159736418

We need an abstraction function that looks something like this:

  >>> assert len(adapter_hexes) == len(vendor_hexes) 
  >>> print adapter_hexes[0]
  0x12345
  >>> groups = get_graphics_device_groups(adapter_hexes, vendor_hexes)
  >>> print groups[0]
  {'adapter_hex': '0x1234', 'vendor_hex': '0x11111', 'adapter_name': 'Something', 'vendor_name': 'Intel'}

That function should be a mix of reading individual pairs for memcache or bunch of up all the unknown in a call to models.GraphicsDevices().get(...)

Comment 1

[Peter Bengtsson [:peterbe]]

Here's a preliminary gist of what my branch is doing:
https://gist.github.com/peterbe/5a0a26ffd29e31a73265
6 different signature reports. 

"NAMES" is a count of (adapter hex, vendor hex) combos I had in cache. 
"MISSING" is pairs not cached. 
"TOTAL" is NAMES+MISSING
"QUERYING" should be the same as the number of MISSING
"HITS" is how many combos came back. This is sometimes larger than QUERYING because some adapter hexes repeat across different vendors. 
"TOOK" is the number of seconds it took to do the whole thing. 

Note how the number of MISSING is going down from signature to signature. When it reaches 0 the whole query takes very little time. 

It's also interesting to note that it always takes around 1 second independent of how many there are to query. That makes perfect sense because of the network I/O latency from my laptop to the PG server is MUCH slower than the query. 

Also, now I'm going to cache each individual combo for 24 hours so that means we're probably going to benefit a LOT more from this caching. Before it used to be 0 https://github.com/mozilla/socorro/blob/085f61878a4d6ad2384e1314e13a875ed666447e/webapp-django/crashstats/crashstats/models.py#L1695 so we always suffered from a ~1 second delay for the WHOLE signature report because of this graphics device lookup.

Comment 2

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3136

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/a911cff76ea1cb4567fb5c6e9cf2d8d15448e9bd
fixes bug 1228122 - Optimize and abstract graphics devices lookup

https://github.com/mozilla/socorro/commit/adba7b322edb1545156e570d3c2d865825d146e3
Merge pull request #3136 from peterbe/bug-1228122-optimize-and-abstract-graphics-devices-lookup

fixes bug 1228122 - Optimize and abstract graphics devices lookup