1160257 - EsRejectedExecutionException when bulk indexing on -stage

Status: RESOLVED WORKSFORME

(support.mozilla.org :: General, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Ever since we updated the ES PHX dev cluster to 1.2.4 (which SUMO -dev and SUMO -stage use), we've been getting this error when bulk indexing:

https://errormill.mozilla.org/support/sumo-stage/group/175654/

(('48 document(s) failed to index.', [{u'index': {u'status': 503, u'_type': u'users_profile', u'_id': u'294066', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@528ec562]',....)

This bug covers figuring out whether that's still happening and fixing it.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Relevant comments from bug #1071044 where it was discussed:

https://bugzilla.mozilla.org/show_bug.cgi?id=1071044#c23

https://bugzilla.mozilla.org/show_bug.cgi?id=1071044#c24

https://bugzilla.mozilla.org/show_bug.cgi?id=1071044#c26

https://bugzilla.mozilla.org/show_bug.cgi?id=1071044#c28

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Also, we should do this soon since if we're still hosed, then this needs to block bug #1160189.

Comment 3

[Michael Cooper [:mythmon]]

This is important. Tossing it in the current sprint as a P1.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Ricky kicked off a reindexing on -stage. It took 2000 years to complete, but a future descendent of mine had the courtesy of hopping in a time travel device and leaving me a post-it note on my monitor that said, "IT WORKED."

I checked Sentry and didn't see any errors. Seems like whatever was causing this problem went away or something.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this 1 point since we spent time talking about it, the ES upgrade for prod, finding all the ES related bugs, re-figuring out the current status of things and then testing this situation out with an indexing run.

Everything seems fine now, so I'm closing this out as WORKSFORME.