Upgrade elasticsearch libraries and code to support ES 5.3

REOPENED
Unassigned

Status

Socorro
General
REOPENED
a year ago
7 months ago

People

(Reporter: adrian, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

a year ago
In order to upgrade our Elasticsearch database to version 5, we need to have code that can support it.

This covers:
* upgrading elasticsearch module to version 5.1
* upgrading elasticsearch-dsl module to version 5.1
* updating all related code to work with ES 5.1 and testing it
(Reporter)

Updated

a year ago
Duplicate of this bug: 1332364
(Reporter)

Comment 3

a year ago
We are going to support version 5.3 instead, since that is now available in AWS.
Summary: Upgrade elasticsearch libraries and code to support ES 5.1 → Upgrade elasticsearch libraries and code to support ES 5.3

Comment 4

a year ago
Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/0dd2f213b73c10a1e7a4bcc309e461f4d35dc84d
[DNM] Fixes bug 1342081 - Upgraded ES libraries and related code. (#3676)

* Fixes bug 1342081 - Upgraded ES libraries and related code.
This commit upgrades our ES modules, elasticsearch and elasticsearch-dsl, as well as all code related to them. It makes sure that unit tests pass using ES 5.1 as the database.

* Trying to use new ES version in travis.

* Try again.

* No more config.
ES 5 does not allow to set index settings in configuration anymore. It has to be done for each individual index. I think we do not really care about that.

* Tentativelly trying a sudo.

* See if that is necessary or not.

* Enabling sudo.

* Using instructions from Peter.

* review comments addressed

* Applied PEP8 and flake8 to the util.py file.

* Moved flatten function to the lib module.

* Adjusted some field mappings for tests to better reflect reality.

* Updated default Super Search Fields list from prod data, for ES 5.1.

* Fixed many schema problems.

* Added unit tests for the __null__ operator.

* Fixed failing test after rebase.

* Updated requirements to use Elasticsearch 5.3.

* Updated travis and circle to run with latest Elasticsearch.

* Added remote_type field to stay up-to-date with prod.

* Fixed an issue in the mapping.

Updated

a year ago
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED

Comment 5

a year ago
Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/58269ed6487471a249b31c0cede4fd1ba46f319c
Bug 1342081 - Revert all ES 5.x code (#3854)

* Revert "[DO NOT MERGE] fixes bug 1379662 - add TelemetryEnvironment to super_search_fields.json (#3848)"

This reverts commit 592b6521c8f40d00530e3ff2ed51d559af543fd0.

* Revert "Bug 1100354 - Use proper way of loading a file. (#3845)"

This reverts commit c7b3e6c4717f381ac0b2b1c7bd59e98ddd1ccaad.

* Revert "Fixes bug 1100354 - Read Super Search Fields from a json file (#3844)"

This reverts commit 673224a186794ff51453a614310555158cb87f55.

* Revert "[DNM] Fixes bug 1342081 - Upgraded ES libraries and related code. (#3676)"

This reverts commit 0dd2f213b73c10a1e7a4bcc309e461f4d35dc84d.
Reopening this so we can close it again later!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 8

11 months ago
Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/bfd0fa286962f964bc70d72797adc0604dc480d9
fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II) (#3935)

* fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II)

* revert socorro.lib.util.flatten

* set BOTO_CONFIG in .travis.yml
We just re-landed this. Marking as FIXED again.
Status: REOPENED → RESOLVED
Last Resolved: a year ago11 months ago
Resolution: --- → FIXED
Created attachment 8901270 [details] [review]
pr 3941 - revert ES 5 changes (2017-08-25)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 11

11 months ago
Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/38cf7c7b6920d39a09037e7960715d950f679ae7
bug 1342081 - Revert es5 (#3941)

* Revert "Bug 1390880 add accessibilityinprocclient and accessibilityclient to super search fieldsjson (#3939)"

This reverts commit c411bd2a2ceb3aac23a61e66167027fb5b4a2b7b.

* Revert "Update ES version in docs"

This reverts commit f3e61dae8c690673d3856889f8c604d1a3d1b232.

* Revert " fixes bug 1381877 - ignore winsock_lsp gt 30000 (#3938)"

This reverts commit 8bca718c1d08e7859e7178a79bebca5851865788.

* Revert "Bug 1393181 dont cache supersearch fields (#3937)"

This reverts commit d6ecf522f4f93352b0302bc149bd599238b6451c.

* Revert "Reapply ES migration related commits (#3936)"

This reverts commit e0174c8448a78af8bd3fe515a580c09be90237fe.

* Revert "fixes bug 1342081 - Upgrade elasticsearch to support ES 5.3 (take II) (#3935)"

This reverts commit bfd0fa286962f964bc70d72797adc0604dc480d9.
No longer blocks: 1100354
(Reporter)

Updated

7 months ago
Assignee: adrian → nobody
You need to log in before you can comment on or make changes to this bug.