948920 - Update Socorro elasticsearch to version 0.90

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

Please update elasticsearch to the latest supported version (0.90.x) for our two clusters, starting with stage so we can verify that the new version doesn't break anything. 

Documentation is here: https://mana.mozilla.org/wiki/display/websites/crash-stats.mozilla.com+%28Socorro%29#crash-stats.mozilla.com%28Socorro%29-InfrastructureOverview

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Up! We should really, really be using 0.90 by now. Can anyone please take care of this? 

Note: it might also be worth going straight to 1.0 or 1.1.

Comment 2

[Daniel Maher [:phrawzty]]

(In reply to Adrian Gaudebert [:adrian] from comment #1)
> Note: it might also be worth going straight to 1.0 or 1.1.

We need to go to 0.90.x before we can go to 1.x, so this step is necessary.  There's no reason for us to *stay* on 0.90.x if we don't want to, but it has to happen nonetheless.

07:42:16 < adrian> phrawzty: can we do it on Tuesday?
07:42:29 < phrawzty> adrian: sure

Tuesday in this case being 22 April 2014.

Comment 3

[Daniel Maher [:phrawzty]]

(In reply to Daniel Maher [:phrawzty] from comment #2)
> Tuesday in this case being 22 April 2014.

To clarify, I'm talking about Stage here - we'll do Prod once we're confident with how Stage went.

Comment 4

[Daniel Maher [:phrawzty]]

Stage went well enough (though it was not without incident):
* The 0.90.10-2.el6 package was built with incorrect permissions. This caused problems when trying to start up again as the permissions needed to be fixed manually.
* The cluster came back up Yellow; it took some eight hours to fully initialize and assign all of the replica shards.

Regarding the latter, this behaviour may be "normal" (i.e. ES working through internal processes to upgrade the indices), or it could have been caused by the tempestuous restarts related to permission issues. Either way, the cluster was still operational, and we did not experience data loss (so no harm no foul).

I have packaged 0.90.13-1.el6 with the correct permissions. :adrian and I will roll it out to production now.

Comment 5

[Daniel Maher [:phrawzty]]

Prod Elasticsearch has been ugpraded to 0.90.13.  It went much more smoothly this time.  The cluster is currently in Yellow, but this is due (as suspected) to normal internal processes due to the upgrade itself - the search service itself is operational.