810540 - Diagnose ElasticSearch timeouts on Mozillians jenkins builds.

Status: RESOLVED DUPLICATE of bug 803599

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Andrei Hajdukewycz [:sancus]]

We're getting ES timeouts on jenkins builds: https://ci.mozilla.org/job/mozillians/296/console

I've tried reverting the code all the way back to the build previous to that, which was green, but it still failed: https://ci.mozilla.org/job/mozillians/306/console

Thus, I'm at least fairly sure that we didn't cause this string of failures directly with a code change, and I'd like some help trying to figure out what's going on here so we can get it corrected and get our builds back to green.

Thanks!

Comment 1

[Giorgos Logiotatidis [:giorgos]]

I increased the ES timeout in jenkins settings and builds returned to green.

rel commit: https://github.com/mozilla/mozillians/commit/2c654dd60d18856c5a660ef3c998853faa1d0564

Comment 2

[Giorgos Logiotatidis [:giorgos]]

It seems that the problem is back.

E.g. https://ci.mozilla.org/job/mozillians/333/console

"""
...

TimeoutError: Request timed out after 5.000000 seconds
...
"""

Comment 3

[Daniel Maher [:phrawzty]]

This is due to a general resource problem on the Jenkins box.  Basically, Elasticsearch and Jenkins are constantly competing for RAM and CPU, which results in one or the other failing in unpredictable ways.  The plan is to build a new services node / cluster to support Jenkins (bug 811380).

That said, I'm not sure that there's really a good short-term solution for the behaviour you're currently experiencing. :/

Comment 4

[Giorgos Logiotatidis [:giorgos]]

Thanks for the update Daniel!

As you said Jenkins / ES are failing randomly and with some luck and multiple tries we still get to run our tests. I guess this is the short-term "solution" for this problem :)

Comment 5

[Daniel Maher [:phrawzty]]

As per comment #3 and bug 803599, closing as dupe.