1378361 - Update to newer Elasticsearch 5.x

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder, enhancement, P2)

Description

[Ed Morley [:emorley]]

We're currently using 5.3.1, but 5.4.3 is out.

It's probably worth being on as new a version as possible before we start using it in production.

We'll need to update (and keep versions in sync):
* vagrant
* travis
* heroku prototype's elastic.co addon

Comment 1

[Ed Morley [:emorley]]

I've also updated the upstream Travis elasticsearch version, so we should be able to use that instead of having to install Elasticsearch manually in every test run:
https://github.com/heroku/stack-images/pull/63

Comment 2

[Ed Morley [:emorley]]

The comment 1 link should have been:
https://github.com/travis-ci/travis-cookbooks/pull/853

Comment 3

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:elasticsearch-5.5.0 > mozilla:master

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e63c8e7f2f7e9116cd030d24cc5917b6303771bc
Bug 1378361 - Update Elasticsearch from 5.3.1 to 5.5.0

This updates the version used in Vagrant and on Travis to the latest
release. The Heroku addon version will be updated separately to match.

Changes between the two versions:
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.3.2.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.3.3.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.4.0.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.4.1.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.4.2.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.4.3.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.5.0.html

Comment 5

[Ed Morley [:emorley]]

I started the update of the Heroku elastic.co ES addon from 5.3.1 to 5.5.0 too, however it seems to have gotten stuck.

I have sent the following to the elastic.co support email:

"""
Earlier today I used the elastic.co dashboard to upgrade our ES instance (ref "REDACTED") from 5.3.1 to 5.5.0, however both nodes (of the new and old version) are still showing on the overview page:
https://cloud-heroku.elastic.co/#clusters/us-east-1/REDACTED/overview/

And on the logs page there are lots of errors about version mismatches.

In addition, the configuration page no longer says ~"upgrade in progress" and instead seems to suggest the cluster is running 5.3.1 still. Trying to upgrade again by selecting 5.5.0 and pressing save, just results in:
"You cluster is not healthy. Please fix it before attempting to change it."
(side note: typo - s/you/your/)

Could you assist?
"""

Comment 6

[Ed Morley [:emorley]]

Reply from support 12 hours ago (2.5 days, 0.5 working days after I emailed):
"""
Hi there,

Thank you for reaching out to Support.

I have looked at your cluster and can see the issue you describe. I've tried to reapply the upgrade for you however it failed again for some reason. I've engaged the backend Cloud team to investigate the issue and will get back to you as soon as they check your cluster for me.

Thanks
"""

Not heard any more since.

Comment 7

[Ed Morley [:emorley]]

10 hours ago:

"""
Hi there, the Cloud team are still reviewing why your cluster failed to upgrade and are still working on it. Will continue to keep you updated.

Thanks
"""

7 hours ago:
"""
Hi there, our Cloud team has rectified the issue and your node is now successfully upgraded to v5.5.0.

Apologies for the inconvenience.

Cheers
"""

Comment 8

[Ed Morley [:emorley]]

It's now broken again, after I pressed "save" on the prefs page. (I've left it over a day and the cluster is still split.)

This is then blocking new deploys on prototype since we seem to have a hard dependency on ES for even the app to start or run collectstatic:
https://dashboard.heroku.com/apps/treeherder-prototype/activity/builds/33999be5-1e58-4403-bb30-aabeb6d1aed7

Seeing that the feature that uses ES isn't turned on anywhere but the prototype instance I say we just delete the ES addon for now and come back to it later.