1342081 - Upgrade elasticsearch libraries and code to support ES 5.3

Status: REOPENED

(Socorro :: General, task, P3)

Description

[[DEACTIVATED] Adrian Gaudebert]

In order to upgrade our Elasticsearch database to version 5, we need to have code that can support it.

This covers:
* upgrading elasticsearch module to version 5.1
* upgrading elasticsearch-dsl module to version 5.1
* updating all related code to work with ES 5.1 and testing it

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3676

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

We are going to support version 5.3 instead, since that is now available in AWS.

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/0dd2f213b73c10a1e7a4bcc309e461f4d35dc84d
[DNM] Fixes bug 1342081 - Upgraded ES libraries and related code. (#3676)

* Fixes bug 1342081 - Upgraded ES libraries and related code.
This commit upgrades our ES modules, elasticsearch and elasticsearch-dsl, as well as all code related to them. It makes sure that unit tests pass using ES 5.1 as the database.

* Trying to use new ES version in travis.

* Try again.

* No more config.
ES 5 does not allow to set index settings in configuration anymore. It has to be done for each individual index. I think we do not really care about that.

* Tentativelly trying a sudo.

* See if that is necessary or not.

* Enabling sudo.

* Using instructions from Peter.

* review comments addressed

* Applied PEP8 and flake8 to the util.py file.

* Moved flatten function to the lib module.

* Adjusted some field mappings for tests to better reflect reality.

* Updated default Super Search Fields list from prod data, for ES 5.1.

* Fixed many schema problems.

* Added unit tests for the __null__ operator.

* Fixed failing test after rebase.

* Updated requirements to use Elasticsearch 5.3.

* Updated travis and circle to run with latest Elasticsearch.

* Added remote_type field to stay up-to-date with prod.

* Fixed an issue in the mapping.

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/58269ed6487471a249b31c0cede4fd1ba46f319c
Bug 1342081 - Revert all ES 5.x code (#3854)

* Revert "[DO NOT MERGE] fixes bug 1379662 - add TelemetryEnvironment to super_search_fields.json (#3848)"

This reverts commit 592b6521c8f40d00530e3ff2ed51d559af543fd0.

* Revert "Bug 1100354 - Use proper way of loading a file. (#3845)"

This reverts commit c7b3e6c4717f381ac0b2b1c7bd59e98ddd1ccaad.

* Revert "Fixes bug 1100354 - Read Super Search Fields from a json file (#3844)"

This reverts commit 673224a186794ff51453a614310555158cb87f55.

* Revert "[DNM] Fixes bug 1342081 - Upgraded ES libraries and related code. (#3676)"

This reverts commit 0dd2f213b73c10a1e7a4bcc309e461f4d35dc84d.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Reopening this so we can close it again later!

Comment 7

[Peter Bengtsson [:peterbe]]

Second attempt https://github.com/mozilla-services/socorro/pull/3935

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/bfd0fa286962f964bc70d72797adc0604dc480d9
fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II) (#3935)

* fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II)

* revert socorro.lib.util.flatten

* set BOTO_CONFIG in .travis.yml

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

We just re-landed this. Marking as FIXED again.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3941 - revert ES 5 changes (2017-08-25)

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/38cf7c7b6920d39a09037e7960715d950f679ae7
bug 1342081 - Revert es5 (#3941)

* Revert "Bug 1390880 add accessibilityinprocclient and accessibilityclient to super search fieldsjson (#3939)"

This reverts commit c411bd2a2ceb3aac23a61e66167027fb5b4a2b7b.

* Revert "Update ES version in docs"

This reverts commit f3e61dae8c690673d3856889f8c604d1a3d1b232.

* Revert " fixes bug 1381877 - ignore winsock_lsp gt 30000 (#3938)"

This reverts commit 8bca718c1d08e7859e7178a79bebca5851865788.

* Revert "Bug 1393181 dont cache supersearch fields (#3937)"

This reverts commit d6ecf522f4f93352b0302bc149bd599238b6451c.

* Revert "Reapply ES migration related commits (#3936)"

This reverts commit e0174c8448a78af8bd3fe515a580c09be90237fe.

* Revert "fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II) (#3935)"

This reverts commit bfd0fa286962f964bc70d72797adc0604dc480d9.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P2 to do soon.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bumping down to P3 until after we look into bug #1568601.