1191939 - switch from elasticutils to elasticsearch-dsl

Status: RESOLVED FIXED

(support.mozilla.org :: Code Quality, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

ElasticUtils has been deprecated and is no longer in development. The Elasticsearch cluster that SUMO uses has been updated to 1.2.4. It's now time to switch SUMO from the ElasticUtils library to elasticsearch-dsl which is the elastic.co maintained successor to ElasticUtils.

I recently did this work for Input under bug #1170333. We can (hopefully) use that as a template for SUMO.

This bug covers switching libraries.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We want to do this as soon as we can for $reasons. Not only are we using a version of Elasticsearch that's no longer supported, but we can't do any new things with Elasticsearch until we switch to elasticsearch-dsl like aggregations.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

cc:ing cyliang so she can follow along with progress on this.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to work on now.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I've been thinking about this and think it's kind of a big project. In order to improve the likelihood of a successful outcome, I'm going to break it into three parts:

1. go through the existing code and clean it up

This might involve some minor refactoring. The focus of this is to clean it up in such a way that we ditch any code we no longer need and the remaining code is clear in its purpose, documented and tested.

2. go through the existing tests and clean those up

I need to make sure things are well-tested before I go do a major refactoring. This is in part because it's much easier to refactor when things are well-tested and also that as far as I'm aware, there are no specifications that detail how this stuff works. So the test suite is really all I have to go on.

3. switch from elasticutils to elasticsearch-dsl

I don't know much about this or whether I can break it up into smaller projects, yet. I'll find out more as I work through steps 1 and 2.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this depend on fixing the index admin bookkeeping, too.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Unless I write a thunking layer like we did with the Sphinx -> Elasticsearch transition, step 3 is an all-or-nothing step.

Working on it now.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I did a lot of work to prepare for this work, but never got to this work. Honest... I ran out of gas. I... I had a flat tire. I didn't have enough money for cab fare. My tux didn't come back from the cleaners. An old friend came in from out of town. Someone stole my car. There was an earthquake. A terrible flood. Locusts! IT WASN'T MY FAULT, I SWEAR TO GOD!

Unassigning myself so someone else can do this.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

This was addressed in d7d8af1ace49301512659785e87fb3230bd34aa8 in July 2020 where it got switched from elasticutils to elasticsearch-dsl. Given that, I'm switching this to FIXED.

Comment 9

[Tasos Katsoulas [:tasos]]

Much better way of handling this bug. Thanks :willkg!