895998 - Jenkins tests are slow

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Lonnen :lonnen]

Needs some investigation, but I bet we can get a 30% speedup without too much trouble.

Comment 1

[Robert Helmer [:rhelmer]]

Would be nice to more effectively re-use the virtualenv, e.g. don't recreate the whole thing on "make install", just reuse the once we build before. Maybe just by fixing bug 871798

Comment 2

[Peter Bengtsson [:peterbe]]

I see this a lot::

```
...

----------------------------------------------------------------------
Ran 185 tests in 48.230s

OK
nosetests --verbosity 1 --with-xunit
Creating test database for alias 'default'...
Tests finished.
FIN
# alias to webapp-django
Running integration test...
INFO: setting up environment... Done.
INFO: setting up database... Done.
INFO: configuring backend jobs... Done.
INFO: starting up collector, processor, monitor and middleware... Done.
INFO: collector test passed
...
```

After it says "Tests finished", there's a delay in tens of minutes till it says "Running integration test..." and "INFO: setting up environment... Done." etc. 

I get a feeling that running the socorro and the webapp-django tests takes about 1-2% of the time it takes to run the whole script. The rest of the time is spent in pip and majority is in executing the integration tests. 


I suspect

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #2)
> I see this a lot::
> 
> ```
> ...
> 
> ----------------------------------------------------------------------
> Ran 185 tests in 48.230s
> 
> OK
> nosetests --verbosity 1 --with-xunit
> Creating test database for alias 'default'...
> Tests finished.
> FIN
> # alias to webapp-django
> Running integration test...
> INFO: setting up environment... Done.
> INFO: setting up database... Done.
> INFO: configuring backend jobs... Done.
> INFO: starting up collector, processor, monitor and middleware... Done.
> INFO: collector test passed
> ...
> ```
> 
> After it says "Tests finished", there's a delay in tens of minutes till it
> says "Running integration test..." and "INFO: setting up environment...
> Done." etc. 
> 
> I get a feeling that running the socorro and the webapp-django tests takes
> about 1-2% of the time it takes to run the whole script. The rest of the
> time is spent in pip and majority is in executing the integration tests. 
> 
> 
> I suspect

Rerunning pip is definitely a problem (not the only one), see comment 1

Comment 4

[Peter Bengtsson [:peterbe]]

Here's what I get on my fancy macbook with SSD:

$ time ./scripts/monitor-integration-test.sh --destroy
INFO: setting up environment... Done.
INFO: setting up database...ERROR: setupdb_app.py failed, check setupdb.log
INFO: cleaning up crash storage directories
INFO: Terminating background jobs
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

real	1m24.508s
user	0m46.775s
sys	0m8.921s





$ time ./scripts/rabbitmq-integration-test.sh --destroy
INFO: setting up environment... Done.
INFO: setting up database...ERROR: setupdb_app.py failed, check setupdb.log
INFO: Purging rabbitmq queue
INFO: cleaning up crash storage directories
INFO: Terminating background jobs
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

real	2m18.522s
user	0m46.369s
sys	0m8.754s


I don't have elasticsearch on my laptop.

Comment 5

[Peter Bengtsson [:peterbe]]

Curious if this is going to help
https://github.com/mozilla/socorro/pull/1412

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/51edb63001b930994a7ca6a361d96be12f9659a1
bug 895998 - measure which integration tests are slow as hell

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/231f91563347b4b172e5fb601bb23b2994be3647
fixes bug 895998 - lots of timings and removed pip flag --ignore-installed, r=lonnen

Comment 8

[Peter Bengtsson [:peterbe]]

Closed because with removing the `--ignore-installed` it takes 5 minutes instead of 25-30 minutes. 

There might be more we can do later but for now, let's resolve this.