674953 - Optimizing search with ElasticSearch

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

The search with ElasticSearch needs some optimizations and changes to be usable in production. Mainly: 

* Choose the good indexes given the date range ;
* Remove as many things as possible from the "query" block (filters are faster) ;
* Run some tests to eventually find other improvements.

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: ES uses dated indexes + fixed date field + some refactoring

This patch changes several things in ES search: 

* Securing the date so we can not search after the current date ;
* Refactoring the code a bit, having more static functions (easier to unit-test) ;
* Fixing the date field to use when querying (date_processed instead of client_crash_date) ;
* Using the dated indexes from prod (like socorro_110730) when querying, and doing it in a secured manner.

Comment 2

[Lonnen :lonnen]

Done any profiling yet?

Comment 3

[K Lars Lohn [:lars] [:klohn]]

Comment on attachment 549886 [details] [diff] [review]
ES uses dated indexes + fixed date field + some refactoring

this looks adequate. You've still got some long lines, but that isn't a show-stopper.

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

Patch landed on trunk with revision r3319: http://code.google.com/p/socorro/source/detail?r=3319

Comment 5

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Use filters instead of queries

Because filters are better than queries in ES (filters are cached, not queries): 

* Replaced the missing and exists query_string with filters ;
* Rewrote the way versions are handled, using filters instead of a query_string ;
* Removed the now useless code about query_string.

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Use filters instead of queries v2

Because filters are better than queries in ES (filters are cached, not queries): 

* Replaced the missing and exists query_string with filters ;
* Rewrote the way versions are handled, using filters instead of a query_string ;
* Removed the now useless code about query_string ;
* Fixed a typo ;
* Fixed the corresponding unit test.

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

Comment on attachment 551914 [details] [diff] [review]
Use filters instead of queries v2

Review of attachment 551914 [details] [diff] [review]:
-----------------------------------------------------------------

Vacations took Lars away.

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

Last patch landed on trunk with commit r3409: http://code.google.com/p/socorro/source/detail?r=3409