853185 - mdn: add ES settings for dev, stage, and prod settings_local.py

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Luke Crouch [:groovecoder]]

Now that bug 824065 is done and we have pushed ES code, we can do this; slowly, server-by-server, and to verify each server. The settings are:

ES_DISABLED = False
ES_URLS = ['http://{dev,stage,prod es cluster host}:{dev, stage, prod es cluster port}']
ES_INDEXES = {'default': 'main_index'}
ES_INDEX_PREFIX = 'mdn'
ES_LIVE_INDEX = True
ES_INDEXING_TIMEOUT = 30

Let's do the dev server first.

Comment 1

[Daniel Maher [:phrawzty]]

I have added the following lines to settings_local.py on Dev (only) :

# bug 853185
ES_DISABLED = False
ES_INDEXES = {'default': 'main_index'}
ES_INDEX_PREFIX = 'mdn'
ES_LIVE_INDEX = True
ES_INDEXING_TIMEOUT = 30
ES_URLS = ['http://elasticsearch-zlb.dev.vlan81.phx1.mozilla.com:9200']

Comment 2

[Luke Crouch [:groovecoder]]

Something seems not right? https://developer-dev.allizom.org/en-US/ hangs forever now.

curl from the box works though ...

$ curl http://elasticsearch-zlb.dev.vlan81.phx1.mozilla.com:9200
{
  "ok" : true,
  "status" : 200,
  "name" : "elasticsearch2_dev_phx1",
  "version" : {
    "number" : "0.20.5",
    "snapshot_build" : false
  },
  "tagline" : "You Know, for Search"
}

James or Will - do these settings look right?

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

What is https://developer-dev.allizom.org/en-US/ doing? Is it doing a search or something else that's ES-related?

Comment 4

[Luke Crouch [:groovecoder]]

No, even the home page (no ES activity) is borked. :(

Comment 5

[Luke Crouch [:groovecoder]]

The first error message in apache logs is/was:

[Sun Mar 24 09:48:21 2013] [error] [client 10.22.81.213] Script timed out before returning headers: kuma.wsgi

That's well after we deployed the ES_* changes, so maybe it's un-related? Did anything happen on 2013-03-24? Maybe just needs a reboot?

Comment 6

[Daniel Maher [:phrawzty]]

Hello,

Just a quick ping as there hasn't been any movement on this bug in a while.  Is this bug still valid ?  Is there anything I can help with ?

Comment 7

[Luke Crouch [:groovecoder]]

Dev looks good. We're going to iterate on our ES index design before we start using in stage or prod.

Comment 8

[Luke Crouch [:groovecoder]]

We've worked thru some ES index issues on dev. Can we make these changes to the stage server now? Do we have an ES stage cluster?

Comment 9

[Daniel Maher [:phrawzty]]

(In reply to Luke Crouch [:groovecoder] from comment #8)
> Do we have an ES stage cluster?

There is no generic ES stage cluster - functionally speaking, other projects simply use the dev ES cluster for staging as well (using a different index).

Comment 10

[Luke Crouch [:groovecoder]]

Okay. Let's make sure to set ES_INDEX_PREFIX = 'mdnstage' in the settings_local.py on stage so it doesn't re-use dev's index. Maybe we should even make ES_INDEX_PREFIX = 'mdndev' for dev too?

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Luke: That's what we do with both SUMO and Input.

Comment 12

[Daniel Maher [:phrawzty]]

I've set this for dev :
ES_INDEX_PREFIX = 'mdndev'

And this for stage :
ES_INDEX_PREFIX = 'mdnstage'

Comment 13

[Luke Crouch [:groovecoder]]

We need more (all) of the ES_* settings to make stage work.

Comment 14

[Daniel Maher [:phrawzty]]

All of the ES_* settings exist in stage (see bug 824065#c19).

Comment 15

[Luke Crouch [:groovecoder]]

Stage looks good. Let's do prod now.

Comment 16

[Daniel Maher [:phrawzty]]

The following config block has been added on the admin host :

ES_DISABLED = False
ES_INDEXES = {'default': 'main_index'}
ES_INDEX_PREFIX = 'mdnprod'
ES_LIVE_INDEX = True
ES_INDEXING_TIMEOUT = 30
ES_URLS = ['http://elasticsearch-zlb.webapp.scl3.mozilla.com:9200']

You will need to trigger a push to prod for this to become active.

Comment 17

[Luke Crouch [:groovecoder]]

Prod push, reindex, and search looks good. Thanks!