901726 - Unit tests take 10x longer to run on server hardware than they do on a Macbook Air

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Wil Clouser [:clouserw]]

Running Marketplace tests locally on a laptop takes around 7.5 minutes.  Running them on ci.mozilla.org took 71 minutes last time (https://ci.mozilla.org/job/marketplace/buildTimeTrend).  I'm having a hard time figuring out how a laptop is out performing a server anyway, but 10x slower sounds like something is broken.

Rumor in #amo was the ES service was the slow point, but I don't have evidence to back that up.

Any ideas on speeding this up?

Comment 1

[Jason Thomas [:jason]]

ci.mozilla.org instance does not seem to be overloaded, but there are a lot of projects configured. It looks like 3-4 projects are building at any given time. I think we should stand up a marketplace/addons specific jenkins instance. Before I do that I need to do some testing to make sure that would be beneficial.

Comment 2

[Christopher Van Wiemeersch [:cvan]]

We right now have 53 Marketplace failures because elasticsearch on jenkins is failing:

https://ci.mozilla.org/job/marketplace/4654/#showFailuresLink

What's the status of this?

Comment 3

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

Machines have been kickstarted and are up and running, however we are having issues with the jenkins puppet module and getting it to cleanly apply. I'm planning on skipping the puppet module for now and installing fresh this coming monday.

Comment 4

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I've manually installed and configured the jenkins software on jenkins1.addons.phx1.mozilla.com:8080, slave running on jenkins2.a.p.m.c. Please let me or the Marketplace ops guys know if there are any issues.

Comment 5

[Christopher Van Wiemeersch [:cvan]]

Here are the errors I'm seeing in our emails:

Changes:

[xwraithanx] handle repeat postbacks (bug 860118)

------------------------------------------
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on jenkins2.addons.phx1.mozilla.com in workspace /opt/jenkins.master/workspace/marketplace
Checkout:marketplace / /opt/jenkins.master/workspace/marketplace - hudson.remoting.Channel@7bd0b029:jenkins2.addons.phx1.mozilla.com
Using strategy: Default
Last Built Revision: Revision 252c40608e4e067ceea557e3e99bc550600a43e8 (origin/master)
Fetching changes from 1 remote Git repository
Fetching upstream changes from origin
Commencing build of Revision 698afdd4680da0d1c597911eab0f34a3241ec285 (origin/master)
Checking out Revision 698afdd4680da0d1c597911eab0f34a3241ec285 (origin/master)
[marketplace] $ /bin/sh -xe /tmp/hudson1532799757050402065.sh
+ SET_ES_TESTS=1
+ ./scripts/run_mkt_tests.sh mkt
Starting build on executor 2... Mon Sep 9 20:01:05 UTC 2013
Warning: no settings directory specified, using: mkt
Usage: ./run_mkt_tests.sh <name> <settings> --with-coverage
Setup... Mon Sep 9 20:01:05 UTC 2013
rm: cannot remove `./venv/lib/python2.6/site-packages/setuptools/script': No such file or directory
rm: cannot remove `template': No such file or directory
rm: cannot remove `(dev).pyc': No such file or directory
rm: cannot remove `./venv/lib/python2.6/site-packages/setuptools/script': No such file or directory
rm: cannot remove `template.pyc': No such file or directory
 You are installing a potentially insecure and unverifiable file. Future versions of pip will default to disallowing insecure files.
 You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
 You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
 Could not find a tag or branch 'cf2033f8bf013039d906a3fe5390ca6940503626', assuming commit.
 Could not find a tag or branch '21078009567fb581e462d4b48a819594f17c5f6f', assuming commit.
 Could not find a tag or branch '54ed3ccb0711b334711dfa4382db6c9ed1da1dff', assuming commit.
 Could not find a tag or branch '27646ecaca40a89024cc581c3ecf5eb0fa87ee11', assuming commit.
 Could not find a tag or branch 'a1ba6ae41bd86f5bb9ff66fb56614e0fafe6e022', assuming commit.
 Could not find a tag or branch '546f339308103880c823b2056830fcdc9220edd0', assuming commit.
 Could not find a tag or branch '729612c2a824a7e8283d416d2084bf506c671e24', assuming commit.
 Could not find a tag or branch 'cb911038e20a399aeff871e199c8bc2be12b5cee', assuming commit.
 Could not find a tag or branch '96e80a64aa4bfcfef4f43fc3ab6966450ccd7325', assuming commit.
 Could not find a tag or branch 'ce5136a257cd44a1c663319124a255c1d10a9834', assuming commit.
 Could not find a tag or branch '36ef06539d6b34c4f345fd0d3e16937d0db9a752', assuming commit.
 Could not find a tag or branch 'a8bd730b202391c080113d224d223463e03088e9', assuming commit.
 Could not find a tag or branch 'f00ad913c62e139d36078e8a7e07dab65a021386', assuming commit.
 Could not find a tag or branch 'fa28ee', assuming commit.
 Could not find a tag or branch 'b0f468dcf33439488158c845df37ef3261852b55', assuming commit.
 Could not find a tag or branch '029dd1263794ec36c327617cd6c2346da81c8c33', assuming commit.
 Could not find a tag or branch 'e2b60a1f96e4c4aed736395c01bf707e969d8e83', assuming commit.
 Could not find a tag or branch 'b4ecca1066bcd1eac87358b4c82902442d4eedc4', assuming commit.
 Could not find a tag or branch 'ce2eba049a5146918fa5239d827bdde9e36eadb7', assuming commit.
 You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
 You are installing a potentially insecure and unverifiable file. Future versions of pip will default to disallowing insecure files.
Command /opt/jenkins.master/workspace/marketplace/venv/bin/python -c "import setuptools;__file__='/opt/jenkins.master/workspace/marketplace/venv/build/MySQL-python/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-9eZKg8-record/install-record.txt --single-version-externally-managed --install-headers /opt/jenkins.master/workspace/marketplace/venv/include/site/python2.6 failed with error code 1 in /opt/jenkins.master/workspace/marketplace/venv/build/MySQL-python
Storing complete log in /var/lib/jenkins/.pip/pip.log
Updating vendor...
Updating product details...
Traceback (most recent call last):
 File "manage.py", line 34, in <module>
   from django.core.management import (call_command, execute_manager,
ImportError: No module named django.core.management
Starting tests... Mon Sep 9 20:03:16 UTC 2013
Traceback (most recent call last):
 File "manage.py", line 34, in <module>
   from django.core.management import (call_command, execute_manager,
ImportError: No module named django.core.management
Build step 'Execute shell' marked build as failure
Recording test results

Comment 6

[Wil Clouser [:clouserw]]

This bug was marked fixed a couple days ago, but https://ci.mozilla.org/job/marketplace/buildTimeTrend says the latest build took 63 minutes.  Reopening - what am I missing?

Comment 7

[Jason Thomas [:jason]]

The new jenkins servers are up. I am still testing to see if there are any performance gains.

Comment 8

[Jason Thomas [:jason]]

The new amo/marketplace jenkins is ready for use at https://ci-addons.allizom.org. I imported amo and marketplace related jobs from ci.mozilla.org over to this new instance. 
amo-master builds in ~25 minutes, marketplace builds in ~20 minutes.

marketplace builds are failing a few tests, could someone please take a look? [1]

Let me know if there are any questions or issues.

[1]https://ci-addons.allizom.org/job/marketplace/lastCompletedBuild/testReport/

Comment 9

[Jason Thomas [:jason]]

The babel related test failures have been fixed by adding 'LC_CTYPE=en_US.UTF-8' to the execute shell command for the marketplace job (thanks @robhudson!). There are currently 3 failing tests for marketplace.

When do we want to switch over to using this instance instead of ci.mozilla.org?

Comment 10

[Andy McKay]

(In reply to Jason Thomas [:jason] from comment #9)
> When do we want to switch over to using this instance instead of
> ci.mozilla.org?

Now would work for me.

Comment 11

[Rob Hudson [:robhudson]]

I agree. Now works now. We can fix the last test or two soon.

Comment 12

[Jason Thomas [:jason]]

I disabled amo/marketplace related jobs on ci.mozilla.org. I enabled email/irc notification on ci-addons.allizom.org. Let me know if there are any issues.

Comment 13

[Wil Clouser [:clouserw]]

I love that we got the build down to 20 minutes, but we're still talking about it taking 3x as long on hardware emphasizing processing speed over hardware emphasizing portability and power conservation.  Have we identified the reason for that difference at least?

Comment 14

[Jason Thomas [:jason]]

The marketplace tests took around 14 mintutes (830 seconds) on my MBP. The MBP memory and disk performs better than the servers, CPU performance is slightly better on the server. 

The tests also don't take advantage of the multiple CPU cores on the server. Are we able to run any of these tests in the suite in parallel?

Comment 15

[Wil Clouser [:clouserw]]

(In reply to Jason Thomas [:jason] from comment #14)
> The marketplace tests took around 14 mintutes (830 seconds) on my MBP. The
> MBP memory and disk performs better than the servers, CPU performance is
> slightly better on the server. 
> 
> The tests also don't take advantage of the multiple CPU cores on the server.
> Are we able to run any of these tests in the suite in parallel?

No, that's bug 794438 and was wontfixed.  Basically running in parallel would require very careful test crafting since any db writes could affect other tests.

Comment 16

[Jeremy Orem [:oremj]]

Given the MBP has much faster disk than the server grade hardware, it seems reasonable that tests on the server hardware are 6 minutes slower.

Parallel tests would be the way to go here. Since each test has side effects we can't just run them all at once and expect the results to be the same each time, as you mentioned.

One quick solution is to run each test suite in parallel, and run each test within the suite sequentially. A new database should be created for each test suite, so that they don't affect each other.

Does that sound doable?

Comment 17

[Wil Clouser [:clouserw]]

We talked about that too when we were talking about parallel tests, but that seemed like maybe more work than it was worth, plus more churn on the db side so we weren't sure if it would be as helpful as we wanted.