809449 - Implement a way for jenkins to override the search index name used in tests

Status: RESOLVED FIXED

(support.mozilla.org :: General, defect, P2)

Description

[Ricky Rosario [:rrosario, :r1cky]]

Right now the search index name is for testing is set in settings_test.py:

    ES_INDEXES = {'default': 'sumo_test'}
    ES_WRITE_INDEXES = ES_INDEXES


The problem is that if you have more than one kitsune suite in jenkins, they will fight each other for this index. We need a way to either make this value dynamic or configurable.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

So, to clarify, those index names get appended to the ES_INDEX_PREFIX. This all happens in apps/search/es_utils.py in READ_INDEX and WRITE_INDEX.

Either turn those "constants" into functions (which needs to be carried through the codebase) or tweak the values to take build.sh arguments into account.

Comment 2

[Michael Cooper [:mythmon]]

Pull: https://github.com/mozilla/kitsune/pull/964

Landed in: https://github.com/mozilla/kitsune/commit/60e1a64950998e24a475d06ced2382c09ac9962f.

This only affects Jenkins, so there is no need to push this to production now.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Seems like we're still having problems with this and that it wasn't fixed. If you run the tests locally, then the index it should be using is "sumo-testsumo" (that's "sumo-test" + ES_INDEX_PREFIX), but it's not:

(M=9d7bd settings_test.py sumodump.sql todo.org) saturn ~/mozilla/kitsune> ./manage.py etusat
09:33:05 INFO    : Settings:
09:33:05 INFO    :   ES_HOSTS              : ['127.0.0.1:9200']
09:33:05 INFO    :   ES_INDEX_PREFIX       : sumo
09:33:05 INFO    :   ES_LIVE_INDEXING      : True
09:33:05 INFO    :   ES_INDEXES            : {'default': 'sumo_dev'}
09:33:05 INFO    :   ES_WRITE_INDEXES      : {'default': 'sumo_dev'}
09:33:05 INFO    : Index stats:
09:33:05 INFO    :   List of indexes:
09:33:05 INFO    :     testindex           : 2 
09:33:05 INFO    :     sumo_test           : 0                    <--
09:33:05 INFO    :     fooindex            : 5 
09:33:05 INFO    :     sumo_sumo_dev       : 3552 READ/WRITE
09:33:05 INFO    :   Read index (sumo_sumo_dev):
09:33:05 INFO    :     questions_question  : 0
09:33:05 INFO    :     forums_thread       : 0
09:33:05 INFO    :     wiki_document       : 3552
09:33:05 INFO    :   Write index is same as read index.


When I r+d the pull request, I didn't test to make sure that it was working correctly--just that the tests ran. My bad.

Anyhow, reopening and grabbing this to look into it further because it's a pain in the ass and screwing us up.

Comment 4

[Michael Cooper [:mythmon]]

The changes I implemented before mainly dealt with the build process that happens on Jenkins, not the tests we run locally. It made changes to the local_settings.py that is generated for Jenkins by build.sh.

I'd love it if you could figure something out something about this, but just a heads up on what the previous change did.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm going to (ab)use this bug to fix the TIME_SLEEP stuff, too. That affects jenkins as well, so it's vaguely related.

http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.cluster_health

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

https://github.com/mozilla/kitsune/pull/1094

That covers everything except the timesleep stuff. I'll do that under a different bug.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I looked into adding a cluster_health call, but pyes' ES class has a refresh method that already does a cluster_health call. So... I'm just going to leave it like that.

Got the r+. Landed the changes in and around https://github.com/mozilla/kitsune/commit/0025b4ca62bc658f115e694f09b6e79e4fa410e0

Going to mark this FIXED. I verified that it's correct when Jenkins tested the pull request.