897541 - test suites taking forever on ci.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[Ricky Rosario [:rrosario, :r1cky]]

At some point in the past two months, our (SUMO) test suite switched to use a separate ES cluster. This improved out times a bunch and our test suite was taking around 5-10minutes reliably. In the past month, something happened and now our tests times are all over the place (depending on load? or?). I've seen them take up to about an hour. The last 5 tests were 42, 24, 16, 23, and 22 minutes.

:groovecoder told me that they are seeing the same thing for mdn.

Can we figure out what happened and fix this? It really affects our dev process since we do continuous deployment.

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

You can see all our build times here:
https://ci.mozilla.org/job/sumo-master/buildTimeTrend

Comment 2

[Michael Cooper [:mythmon]]

For a better graph, I grabbed the last 800 builds from the Jenkins API, and graphed them here: http://bl.ocks.org/mythmon/6072959.

That shows each build, and you can hover over it to get the date, and it also shows a 30 build moving average , which shows that average response times have been going up lately.

Comment 3

[Michael Cooper [:mythmon]]

More graphs: http://bl.ocks.org/mythmon/raw/6073408/

This shows 5 projects, and the trend of increasing build times is evident in all of them, except socorro-crashstats. socorro-crashstats does not (I am told) use a database, so my theory is that the database is what is causing the slowness here.

Comment 4

[Brandon Burton [:solarce]]

We're seeing odd slowness from other jobs, such as socorro-*, as well.

I am going to restart Jenkins at 9PM PDT tonight to see if that helps.

Comment 5

[Brandon Burton [:solarce]]

Jenkins and data services restarted at 9:01PM

Jenkins happy at 9:40PM

Let's see how your builds go over the next couple of days

Comment 6

[Brandon Burton [:solarce]]

Any better?

Comment 7

[Luke Crouch [:groovecoder]]

2 builds on mdn were 7m then 12m
8 builds on mdn-github were about the same