682140 - Filtered results for ES on khan don't match results on prod

Status: RESOLVED INVALID

(Socorro :: General, task)

Description

[Matt Brandt [:mbrandt]]

khan returns 24 results whereas production returns 11.

The filtered results don't match
http://agaudebert.khan.mozilla.org/query/query?product=Firefox&version=Firefox:8.0a1&range_value=1&range_unit=days&date=08/14/2011+00:00:00&query_search=signature&query_type=exact&query=mozalloc_abort+|+NS_DebugBreak_P+|+X11Error&reason=&build_id=&process_type=any&hang_type=any&do_query=1

and

https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox:8.0a1&range_value=1&range_unit=days&date=08/14/2011+00:00:00&query_search=signature&query_type=exact&query=mozalloc_abort+|+NS_DebugBreak_P+|+X11Error&reason=&build_id=&process_type=any&hang_type=any&do_query=1

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

This is due to the way I filter products with ElasticSearch. We get results from the product called Firefox-Trunk when searching for Firefox only.   

The best (and only, so far) way to solve this would be to change the mapping of the 'product' field to add a non-analyzed field, so I can easily search for 'Firefox' only. 

Anurag, I didn't find any other solutions to solve this, and I believe that re-indexing, even if it's a pain, is the easiest and cleanest solution we have. I couldn't test it either so if you can help me doing it I would really appreciate. To test it I would need an ES instance using the new mapping and containing crashes with product = Firefox and Firefox-Trunk. 

If you can help me with setting this up, it won't take too long for me to change my code and make sure it works. If you cannot do it, I will try to find some time over the weekend to do it, and I'll let you know how it goes.

Comment 2

[Anurag Phadke[:[e-mail]]]

@adrian: i can start the reindexing in couple of hours.

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

@aphakde: As I told you I'm out for the weekend, so do it whenever it's easiest for you. Once it's done I'll fix the search and make a patch.

Comment 4

[Robert Kaiser]

(In reply to Adrian Gaudebert [:adrian] from comment #1)
> This is due to the way I filter products with ElasticSearch. We get results
> from the product called Firefox-Trunk when searching for Firefox only.  

Yup, with product=Firefox, getting those "Firefox-Trunk" things is unexpected. Without a product in the query, it would be fine.

Comment 5

[Anurag Phadke[:[e-mail]]]

@adrian:
indexed 10m docs so far, should be done by 2nite. 
                    "product": {
                        "type": "string",
                        "index": "not_analyzed"
                    },

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Use not analyzed product field in ES

This patch is very very small as the main work to resolve this bug was done directly on ElasticSearch: Anurag changed the mapping so the product field is both analyzed and not analyzed, which allows me to filter by this entire field instead of by terms. This way, Firefox-Trunk-like results will no more appear when asking for Firefox only.

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

Comment on attachment 556996 [details] [diff] [review]
Use not analyzed product field in ES

Review of attachment 556996 [details] [diff] [review]:
-----------------------------------------------------------------

Ask Brandon to review.

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

Last patch landed on trunk with commit r3485: http://code.google.com/p/socorro/source/detail?r=3485

Comment 9

[Matt Brandt [:mbrandt]]

Sadly reopening, this query 500's:

Steps to reproduce:
http://agaudebert.khan.mozilla.org/query/query?product=Firefox&version=Firefox:8.0a1&range_value=1&range_unit=days&date=08/14/2011+00:00:00&query_search=signature&query_type=exact&query=mozalloc_abort+|+NS_DebugBreak_P+|+X11Error&reason=&build_id=&process_type=any&hang_type=any&do_query=1


Results in:

[11:17:59.423] GET http://agaudebert.khan.mozilla.org/query/query?product=Firefox&version=Firefox:8.0a1&range_value=1&range_unit=days&date=08/14/2011+00:00:00&query_search=signature&query_type=exact&query=mozalloc_abort+|+NS_DebugBreak_P+|+X11Error&reason=&build_id=&process_type=any&hang_type=any&do_query=1 [HTTP/1.1 500 Internal Server Error 67ms]

Comment 10

[Matt Brandt [:mbrandt]]

Thanks Adrian, this is now working correctly on your khan instance. Let's wait until this properly lands on dev/stage and then we'll bump it to verified.

Comment 11

[[DEACTIVATED] Adrian Gaudebert]

We're waiting for a link to be opened between dev/stage and ES to be able to QA it.

Comment 12

[[DEACTIVATED] Adrian Gaudebert]

This is old and probably invalid now. We'll open a new bug if we hit this problem again.