999066 - intermittent test failure in elastic search tests

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Lonnen :lonnen]

We've started having an intermittent test failure originating in ES tests. If there isn't an obvious quick fix we should disable the test.

00:00:54.857 env  resource.postgresql.database_hostname=jenkins-pg92  secrets.postgresql.database_username=test  secrets.postgresql.database_password=aPassword  resource.postgresql.database_port=5432  resource.rabbitmq.host=rabbitmq-zlb.webapp.phx1.mozilla.com  resource.rabbitmq.virtual_host=socorro-jenkins  secrets.rabbitmq.rabbitmq_user=socorro-jenkins  secrets.rabbitmq.rabbitmq_password=aPassword  elasticsearch_urls=http://jenkins-es20:9200  elasticSearchHostname=jenkins-es20 PYTHONPATH="." /var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/bin/coverage run /var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/bin/nosetests socorro -s --with-xunit
00:01:00.403 ..............................................................E...........................................................................................................................................................................................................................................................................S..............S............S.............................................................................................................S.................................................................................................................................................................................................................................................................................................................................
00:09:53.655 ======================================================================
00:09:53.655 ERROR: test_right_indices_are_deleted (socorro.unittest.cron.jobs.test_elasticsearch_cleanup.IntegrationTestElasticsearchCleanup)
00:09:53.655 ----------------------------------------------------------------------
00:09:53.655 Traceback (most recent call last):
00:09:53.655   File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro/unittest/cron/jobs/test_elasticsearch_cleanup.py", line 89, in test_right_indices_are_deleted
00:09:53.655     'alias': last_week_index
00:09:53.655   File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 96, in decorate
00:09:53.655     return func(*args, query_params=query_params, **kwargs)
00:09:53.655   File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 720, in update_aliases
00:09:53.655     body=settings, query_params=query_params)
00:09:53.655   File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 254, in send_request
00:09:53.655     self._raise_exception(resp, prepped_response)
00:09:53.655   File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 269, in _raise_exception
00:09:53.655     raise error_class(response.status_code, error_message)
00:09:53.655 ElasticHttpError: (400, u'InvalidAliasNameException[[socorro_some_aliased_index] Invalid alias name [socorro201415], an index exists with the same name as the alias]')
00:09:53.655 
00:09:53.658 ----------------------------------------------------------------------
00:09:53.658 Ran 790 tests in 533.326s
00:09:53.658 
00:09:53.658 FAILED (SKIP=4, errors=1)

Comment 1

[Lonnen :lonnen]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/9befc733f365eddd043c2e30d06c3d552e340b57
Fixes bug 913039 - Added a cron job to delete old elasticsearch indices. r=peterbe,phrawzty

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/aad833ab9b686f7cf9891c161b242c2d89182791
Fixes bug 999066 - Ensured all elasticsearch indices are removed after tests run.