678107 - Test search performances with large number of results from ElasticSearch

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

We should make sure that asking for a very large amount of data (in our case, distinct signatures) has an impact on performances with ElasticSearch. Should be tested both from the UI and directly on the server side.

If performances appear to have a real impact, we should limit the number of results we get.

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Timing queries depending on number of results

Here is the result of my performance tests. What I call the "number of results" is the number of distinct signatures I ask during the first query a search runs (the second one being doing facets on the first 100 signatures). 

It is clear from those results that asking for a large number of results when there is a large number of results is slow. Limiting this number to 1,000 is particularly efficient for huge queries (see the first one, a month of data, product = firefox, os = all). 

Anurag, Daniel, do you think this is enough to say that that specific number is a point of failure and we should do something about it, or do you want to run more tests?

Comment 2

[Anurag Phadke[:[e-mail]]]

 i would say that fetching more than 1K reuslts in single query should be avoided as far as possible coz u r not only hitting CPU/RAM but also network traffic for such large datasets and run the risk of crashing the client too

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

I think that these timing results definitely seem to point at using a limit of 100 or 1000 to be a very good thing to keep the website feeling responsive and save load on the search cluster.

I think we should push this to socorro-dev and see if Kairo and Laura agree to make it a limitation to the ES functionality when released.

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

Some ideas to solve this problem: 

* Limiting the maximum number of distinct signatures to 1,000. Period. 
* Limiting the maximum number of distinct signatures to the number of results needed for the current request (= page offset + number of results per page) plus 1,000. This way if the user wants to see all the results he can (even if it will be long to get to page 300), and we reduce the number of processed signatures for the majority of searches. 

Any other idea would be appreciated.

Comment 5

[Laura Thomson :laura]

Do this:
* Limiting the maximum number of distinct signatures to 1,000. Period.

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

Option 1 is implemented and can be tested on khan: http://agaudebert.khan.mozilla.org/query

Maybe we should add a little something to say that we limit the number of results to 1,000? Like "1,000+ Results"?

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Limit number of results to 1,000

This patch adds a new configuration key, and limit the number of results we get from ES to this number. By default the number is 1,000.

Comment 8

[Lonnen :lonnen]

Comment on attachment 554261 [details] [diff] [review]
Limit number of results to 1,000

'limitate' should be limits in the elasticsearch.py changes, and in webapiconfig doc string you're adding "can" might be more accurate as "should".

Fix those and this is r+.

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

Last patch (with fixed typos) landed on trunk with commit r3473: http://code.google.com/p/socorro/source/detail?r=3473