1073553 - No well-defined or documented way to run unittests for build/tools repository locally

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Massimo Gervasini [:massimo]]

Today I was testing a patch on the tools repository. 

I have tried to execute the tools tests on my local machine in several ways, each execution reported a different number of tests:

 number of tests |  command 
 ----------------+---------------
        269      |  python -m unittest discover -s lib/python
        338      |  tox
        N/A      |  python setup.py test **
        307      |  using a local copy of script we are using on Jenkins  

 and tests did not pass, our Jenkins instead, reports 338 tests and no failures.


We should have a consistent way to run tests locally and on Jenkins, and results should be independent from the platform type.


** python setup.py test returns AttributeError: 'module' object has no attribute 'test_release_config', this can be related to my local python setup

Comment 1

[Chris AtLee [:catlee]]

totally agreed!

I think we should use tox to run tests. I've patched up the version of tox.ini we have right now, and it's passing for me (on my laptop!)
http://hg.mozilla.org/users/catlee_mozilla.com/tools/rev/e609241abd0c

I *think* all that's required is to:
mkvirtualenv tox
pip install tox
tox

Comment 2

[Massimo Gervasini [:massimo]]

Attachment: [tools] Bug 1073553 - replace "touch" calls with a pure python function.patch

Hi Chris,

can we port your changes to our tools repository?

With your changes and this patch, I have all tests passing on osx.

Right now, on osx, tests are failing because we call touch -d yesterday, but on osx, touch does not accept the -d option.

Comment 3

[Chris AtLee [:catlee]]

Attachment: fix up tox.ini, make hg tests more reliable

Comment 4

[Massimo Gervasini [:massimo]]

Comment on attachment 8504063 [details] [diff] [review]
[tools] Bug 1073553 - replace "touch" calls with a pure python function.patch

r+ from irc
<mgerva> catlee: if you agree i'm going to land my patch on tools tests - it was submitted just for feedback, not review
<catlee> mgerva: yeah, sounds fine
<catlee> it just affects tests I think
<mgerva> catlee: yes it's only about tests
<catlee> mgerva: yeah, looks fine

Comment 5

[Massimo Gervasini [:massimo]]

Attachment: tox error

When I run tox, from a clean virtualenv, it just works. If I run tox a second time, I get a:

AssertionError: '1.1.2pre' != '1.1.3pre'

(log attached)

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 8517515 [details]
tox error

I've hit this as well, and haven't been able to track down the cause.

Comment 7

[Pete Moore [:pmoore][:pete]]

Hmmm I wonder if this is related...

From https://testrun.org/tox/latest/

> When you run tox a second time you’ll note that it runs much faster because it keeps track of
> virtualenv details and will not recreate or re-install dependencies.

Comment 8

[Pete Moore [:pmoore][:pete]]

I've set this up in travis... Same error as above, but curiously in travis we get this with the first (and only) invocation of tox:

https://travis-ci.org/petemoore/build-tools/builds/41242097

======================================================================

FAIL: testBumpDefaultVersionTxt (mozilla_buildtools.test.test_build_versions.TestBumpFile)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/build/petemoore/build-tools/lib/python/mozilla_buildtools/test/test_build_versions.py", line 214, in testBumpDefaultVersionTxt

self._doTest("default-version.txt", "1.1.2pre", "1.1.3pre", "1.1.3pre")

File "/home/travis/build/petemoore/build-tools/lib/python/mozilla_buildtools/test/test_build_versions.py", line 192, in _doTest

self.assertEquals(newContents, expectedContents)

AssertionError: '1.1.2pre' != '1.1.3pre'

----------------------------------------------------------------------

Ran 345 tests in 199.266s

FAILED (failures=1)

ERROR: InvocationError: '/home/travis/build/petemoore/build-tools/.tox/py27-hg2.6/bin/coverage run --branch --source /home/travis/build/petemoore/build-tools/lib/python /home/travis/build/petemoore/build-tools/.tox/py27-hg2.6/bin/nosetests --with-xunit /home/travis/build/petemoore/build-tools/lib/python'

___________________________________ summary ____________________________________

ERROR: py27-hg2.6: commands failed

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: bug1073553_tools_v1.patch

This patch sets up the tox testing in travis.

Comment 10

[Massimo Gervasini [:massimo]]

Comment on attachment 8523840 [details] [diff] [review]
bug1073553_tools_v1.patch

>diff --git a/.travis.yml b/.travis.yml
>new file mode 100644
>index 0000000..00ab04b
>--- /dev/null
>+++ b/.travis.yml
>@@ -0,0 +1,9 @@
>+language: python
>+python:
>+  - "2.7"
>+
>+install:
>+  - "pip install tox"
>+
>+script:
>+  - bash ./run_travis_tests.sh
>diff --git a/run_travis_tests.sh b/run_travis_tests.sh
>new file mode 100644
>index 0000000..1cf9452
>--- /dev/null
>+++ b/run_travis_tests.sh
>@@ -0,0 +1,16 @@
>+#!/bin/bash
>+
>+set -xveu
>+
>+echo "Setting up git environment..."
>+git config --global user.email "[e-mail]"
>+git config --global user.name "Travis McTravis"
>+
>+echo "Setting up hg environment..."
>+cat << EOF > ~/.hgrc
>+[ui]
>+username = Travis McTravis <[e-mail]>
>+EOF
>+
>+echo "Running tests..."
>+tox


In theory this works but I've found some issues:
* running run_travis_tests.sh overwrites your local user setting
* we already have an hgrc file we use for test
* there's no need for run_travis_tests.sh if we update .travis.yml to something as:

language: python
python:
  - "2.7"

before_script:
  - git config --global user.email "[e-mail]"
  - git config --global user.name "Travis McTravis"
  - cp lib/python/mozilla_buildtools/test/hgrc ~/.hgrc

install:
  - "pip install tox"

script:
  - "tox"

What do you think, Pete? 

PS Here are the result of travis running with this configuration: https://travis-ci.org/gerva/tools/builds/41260898 (It fails with the same error as in comment #5 and comment #8)

Comment 11

[Pete Moore [:pmoore][:pete]]

Attachment: bug1073553_tools_v2.patch

Thanks for your feedback Massimo, totally agree! New patch attached.

Comment 12

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8524633 [details] [diff] [review]
bug1073553_tools_v2.patch

https://hg.mozilla.org/build/tools/rev/23fe03ae6f32

Comment 13

[Pete Moore [:pmoore][:pete]]

So this is landed now, but we have this one failing test. I will close this bug, as this test failure is a separate issue, which I will raise with the test author in a separate bug.