706258 - elastic timeout errors

Status: RESOLVED DUPLICATE of bug 706944

(support.mozilla.org :: Search, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

pennyfx was talking about how elastic is flakey and querying sometimes times out.  We haven't implemented the query code yet in SUMO, so I don't know if this affects us as well or not.  However, I hit it pretty regularly when building the indexes from scratch.

There was a pull request in elasticutils that would retry queries.  If we have this problem, we should look at that and pull it into the sumo branch of elasticutils.  If not, we should ditch this.

Comment 1

[Erik Rose [:erik][:erikrose]]

Pretty satisfied this is covered by...

* bug 706944 (tuning settings of ES itself)
* bug 712069 (live indexing retrying)
* bug 715932 (full indexing robustness)

Reopen or do something else smart if you disagree!

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

As a side note, we shouldn't add code for retrying queries--ES already does it. I added some code to search/es_utils.py for building our own ES that allows us to adjust the retry amount.

Comment 3

[Erik Rose [:erik][:erikrose]]

Interesting! Link?

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

http://packages.python.org/pyes/references/pyes.es.html#pyes.es.ES

This is where the MaxRetryError comes from.

This retries immediately, so it doesn't cover retrying in, say, 20 minutes. That's covered under bug #712069 and probably something we want to write code to deal with.