1085823 - [Super Search Fields] Turn dynamic mapping on for Elasticsearch

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

The feature landed with bug 1013322, we now need to enable it. Let's first do it on stage, and look at it for a bit, before we turn it on on production. 

To turn it on, the option `use_mapping_file` must be set to `False` in the `processor.ini` config file.

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: mapping_config.diff

Here is the diff that needs to be applied to our svn config repo. Can anyone with write access please apply it for me? Phrawzty maybe?

Comment 2

[Daniel Maher [:phrawzty]]

$ svn ci -m 'enable dynamic mapping; bug 1085823'
Sending        processor.ini
Transmitting file data .
Committed revision 95603.

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

Thanks :phrawzty! We will start seeing this effects of this on Monday, when a new index is created with this dynamic schema. Let's hope it will go well! :)

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

Attachment: mapping_conf_2.diff

It didn't work on stage, the new index still used the mapping from our file system. Looking at the logs, I suspect this is also needed. :phrawzty do you mind applying this to our config repo?

Comment 5

[Daniel Maher [:phrawzty]]

$ svn ci -m 'disable use_mapping_file; bug 1085823'
Sending        files/stage/etc-socorro/common_elasticsearch.ini
Transmitting file data .
Committed revision 96215.

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

This has been working on stage for 2 weeks now, and we didn't notice any problem. The processors have logged some errors that we already had before (due to over-sized numbers), so nothing bad there. 

However, since I will likely be on PTO next week (as most of our US colleagues) and the following week will be MozLandia, I suggest we do not turn this on on production just yet. Let's wait for a time where I can be fully operational on a Monday morning, when the effects of this change will be seen.

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

OK, since this is becoming more and more needed, I want to turn it on asap, so it takes effect starting next Monday. It's been running on stage for a long time, and since several changes have been made in the Super Search Fields recently we now that the mapping is working (it's valid for Elasticsearch and several hundreds of crashes have been successfully indexed with this mapping, as for the tests in place). 

If, for some reason, things go wrong and we lose data in Elasticsearch, here is the process to turn it off: 

1. update configuration of the processor (in `files/prod/etc-socorro/processor.ini`), set `use_mapping_file` option to False
2. stop the processors
3. delete the current index (socorro%Y%W, so it would be socorro201450 next week)
4. restart the processors
5. reprocess all crashes from 00:00:00 UTC to the moment when processors where stopped

:phrawzty, will you be available next Monday (Dec 15) to help me if things go wrong?

:lonnen, do I have your approval?

Comment 8

[Daniel Maher [:phrawzty]]

(In reply to Adrian Gaudebert [:adrian] from comment #7)
> :phrawzty, will you be available next Monday (Dec 15) to help me if things
> go wrong?

I will be available on that date.

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

Index: common_elasticsearch.ini
===================================================================
--- common_elasticsearch.ini	(revision 98104)
+++ common_elasticsearch.ini	(working copy)
@@ -19,6 +19,9 @@
 # a class that will manage transactions
 #transaction_executor_class='socorro.database.transaction_executor.TransactionExecutorWithLimitedBackoff'
 
+# load the mapping from a file if true, load it from the database otherwise
+use_mapping_file=False
+
 
Index: processor.ini
===================================================================
--- processor.ini	(revision 98104)
+++ processor.ini	(working copy)
@@ -230,7 +230,7 @@
 
         # load the mapping from a file if true, load it from the database otherwise
         # see "resource.elasticsearch.use_mapping_file" for the default or override it here
-        #use_mapping_file=False
+        use_mapping_file=False
 
$ svn commit -m 'Bug 1085823 - Turn dynamic mapping on for Elasticsearch in production'
Sending        common_elasticsearch.ini
Sending        processor.ini
Transmitting file data ..
Committed revision 98105.

Comment 10

[[DEACTIVATED] Adrian Gaudebert]

So far this seems to be working well. The few errors that we see in the processors since this morning are errors that we used to see before as well, and unrelated to this change.