Index creation is broken in elasticsearch 5.3

RESOLVED FIXED

Status

Socorro
Processor
RESOLVED FIXED
5 months ago
4 months ago

People

(Reporter: miles, Assigned: adrian)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 months ago
Since we upgraded the processor to work with our elasticsearch 5.3 cluster, it has been failing to process crashes. In meeting, adrian said that each time the processor attempts to process a crash, it attempts to create the index in elasticsearch that that crash should be put into.

We are now in week 27 of the Socorro calendar, and the processor has thus far failed to create a new index. Will determined that [0] occurs in the part of the processor's code that creates new indices. So this bug appears to block all processing.

Will also hypothesized that the code that does index creation in the processor was maybe not updated correspondingly for 5.3.

For some more general info, see the migration status Gdoc [1].

[0] https://sentry.prod.mozaws.net/operations/socorro-stage/issues/619003/
[1] https://docs.google.com/a/mozilla.com/document/d/1dtlZW2xaKy_jHFg9fg0VX2ddpH1eTC3OoWgoRP4WdfM/edit?usp=sharing
(Reporter)

Comment 1

5 months ago
Throwing this your way Adrian.
Flags: needinfo?(adrian)
(Assignee)

Comment 2

5 months ago
I think I know what's going on. I'm guessing you migrated the `socorro` index from the current cluster to the new one? If so, that failure makes sense: the content of the `socorro` index has to be wildly different between the old and new clusters, because they contain data used to generate the mapping of new data indices. 

The solution to this problem is simple: delete the `socorro` index from the new cluster, then run `scripts/setup_supersearch_app.py` from any box that has the correct code and configuration. It will put the right data into the `socorro` index, and then processing should start working again.

I am going to do that now.
Flags: needinfo?(adrian)
(Assignee)

Comment 3

5 months ago
There's a problem with the mapping that ES creates by default for the `socorro` index, that causes a conflict with some of the data we want to have in there. I am currently working on creating a better mapping for that index, that should solve our issues.
Assignee: nobody → adrian
(Assignee)

Comment 4

4 months ago
This is resolved.
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.