If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

[ci.mozilla.org] marketplace jenkins job failing on ES tests due to test_mkt_amo shards being corrupted

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
P3
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: robhudson, Assigned: solarce)

Tracking

Details

(Reporter)

Description

4 years ago
Approximately after the jenkins upgrade some of our tests started failing. I'm looking for more insight into the failures to try to either fix our tests or help configuring things so the tests have what they need to pass?

The test failures are here:
https://ci.mozilla.org/job/marketplace/3917/#showFailuresLink
(Assignee)

Updated

4 years ago
Assignee: server-ops-webops → bburton
Priority: -- → P3
(Assignee)

Comment 1

4 years ago
Due to ES 0.17.4 having health issues, the shards for test_mkt_amo have been corrupted and this is why your ES tests are failing, like

Exception pyes.urllib3.connectionpool.TimeoutError: TimeoutError('Request timed out after 30.000000 seconds',) in <bound method ES.__del__ of <pyes.es.ES object at 0xcbb10d0>> ignored
ERROR
Summary: After upgrade jenkins test failures → [ci.mozilla.org] marketplace jenkins job failing on ES tests due to test_mkt_amo shards being corrupted
(Assignee)

Comment 2

4 years ago
We're in the process of retiring ES 0.17.4, we have an ES 0.20 installation available for tests on a new dedicated server which only runs data servers for Jenkins tests.

In order to use the new instance you need to update your ES tests to use 'jenkins-es20' instead of 'localhost' for your tests configs

It would be ideal if you can make the above change and your builds should start passing again as they'll make a new index the first time they talk to the new ES instance.
Status: NEW → ASSIGNED
(Assignee)

Comment 3

4 years ago
In the mean time I've deleted the corrupt index on 0.17.4

bburton@ironbars [09:47:31] [~]
-> % curl -XDELETE jenkins1.dmz.phx1.mozilla.com:9200/test_mkt_amo/
{"ok":true,"acknowledged":true}%                                                    

And am running https://ci.mozilla.org/job/marketplace/3919/console , this should recreate the 0.17.4 index and i will watch the job and confirm this
(Reporter)

Comment 4

4 years ago
I can work on updating our code to point to the new 0.20 instances. We use 0.19.x in production so this should be a bit closer.
(Assignee)

Comment 5

4 years ago
https://ci.mozilla.org/job/marketplace/3919/console passed.

:robhudson will be switching to 0.20.

ES 0.17 is happy with the test_mkt_amo index as well.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.