1342083 - Create a mapping file that works with Elasticsearch 5.1

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

In order to migrate data from our current ES 1.4 to ES 5.1, we will need to recreate all our indices and reindex all our data. However, the mapping syntax has changed since v1.4, and we won't be able to use the current mapping.

Since we will use reindex-from-remote [0] to move our data, and that module requires us to create indices before we write to them anyway, we can create a temporary mapping JSON document that we can use to create new indices in the new cluster.

[0] https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html#reindex-upgrade-remote

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Here is a schema that I have tested locally and seems to be working as needed: https://raw.githubusercontent.com/adngdb/socorro/ec47703d332ddc83d5f7c20dbd01805207a2fb2c/scripts/data/supersearch_fields.json

We might have to make some adjustments after we are done migrating, but that will be done through our Super Search Fields admin panel.

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

Attachment: es_mapping_v5.3.3-2.json

New mapping that contains the TelemetryEnvironment field.

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

Here's the code I use to generate that mapping: https://gist.github.com/adngdb/fd30965ce5c1e5523f2880d9a33dad20

Process is: make sure you have all the ES 5.x code, make the changes you need to the socorro/external/es/data/super_search_fields.json file and then run the script: 

$ python create_es_index.py > es-mapping.json