707319 - Turn off text analyzers in buildbot ES

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Jonathan Griffin (:jgriffin)]

In the buildbot ES (buildbot-es.sjc1.metrics.mozilla.com), the text analyzers were originally off, but somehow got turned on during an earlier ES upgrade.

The result is that terms like "mozilla-central" are now analyzed as two separate terms, instead of one, and our queries are much easier when "mozilla-central" is indexed as a single term.

We still want date analyzers that turn text fields that look like dates into date objects, but we'd like the text analyzers that split fields into separate terms turned off.

I volunteer to help write scripts to help deal with any data migration that might be necessary because of this.

This isn't an emergency but it would be nice to deal with this sometime this month.

Comment 1

[Anurag Phadke[:[e-mail]]]

@jgriffin: We have upgraded ES to 0.18.6 for the WOO/buildbot cluster. Please hit me up on IRC whenever you have a chance so we can work towards resolving this bug.

Thx,

Comment 2

[Anurag Phadke[:[e-mail]]]

Spoke with kimchy over IRC and we will need to re-index the entire dataset.

Next steps:
1. Find 3 temp machines and install ES-0.18.6 with existing data.
2. add mappings file to "not analyze" fields
3. run re-indexing via https://metacpan.org/module/ElasticSearch#reindex- or equivalent.
4. if everything goes as planned, repeat steps 2-3 for prod cluster
5. create an alias that points to the newly created index.
6. modify existing code to point to the new alias.

Comment 3

[Anurag Phadke[:[e-mail]]]

:jgriffin - what do u suggest we should do? IT now manages ES, should we talk to them?

Comment 4

[Annie Elliott]

migrating to IT

Comment 5

[Sheeri Cabral [:sheeri]]

Removing unrelated cc's and moving to the proper queue.

Comment 6

[C. Liang [:cyliang]]

@jgriffin: Unfortunately, I have no context for this bug about buildbot ES and the need to re-index terms. Two quick questions for you:
   1) is this bug still relevant, and 
   2) if so, do you have some time to discuss time frames for getting this done?  (IRC or email would be fine)   

I'm not sure where the data currently resides since 1) I'm fairly certain that the server mentioned in the initial description is gone and 2) I'm not sure what indexes are being discussed.

Comment 7

[Jonathan Griffin (:jgriffin)]

This bug is obsolete now.