1076810 - RelEng CI tests should be available over the internet, not just from VPN

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

Currently jenkins is hosted on an internal private IP (http://10.134.48.37:8080/). AFAIK there is no natting to expose this publicly.

This means contributors cannot see our CI results, which is a big reason not to contribute!

An alternative to exposing this service publicly, would be to migrate over to travis, for example, which also might save some costs, but also come at some cost to migrate.

I'm also aware other teams than RelEng (such as A Team) rely heavily on mozharness, so they may wish to enter into the discussion of where they would like mozharness CI to live - and of course this is also CI for other repos, like buildbot, tools.

If we do decide to expose it, maybe it should move over to Dev Services management?

If we move to travis, please note there may be a slight further delay (~1 minute) until results are published, because our github repos lag around 1 minute behind our hg repos, due to the time it takes to mirror changes from hg.m.o -> github. However, it considerably increases visibility.

Comment 1

[Armen [:armenzg]]

+1!

I don't find these results critical to get them fast, however, it is nice to make them public.

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_v1.patch

So, out of the box the mozharness tests are not working, but I don't think that should stop the delivery of standing up the CI in travis - rather I'd like to get travis CI enabled for the various build repos, and then run through the issues to fix them.

Check out https://travis-ci.org/petemoore/build-mozharness to see results.

Comment 3

[Pete Moore [:pmoore][:pete]]

I've spotted we have a script in mozharness for running unit tests (https://github.com/mozilla/build-mozharness/blob/master/unit.sh), but I think I still prefer calling nosetests directly that running this script because:

1) the unit test script is also running coverage tests, and afaik, we have nowhere to publish the reports when running from travis, so this could be wasted effort
2) I think it is a fair requirement that the nosetests command should run all the unit tests correctly out-of-the-box, so we shouldn't need a bash script to invoke it which adds other logic too - based on this I'd like to land the current patch, and then work on fixing up the tests so that:
a) all of them run that can run,
b) they all pass

There are also a third set of test results, which we currently get in our jenkins CI (which travis would replace). I've managed to get admin access to the job configuration, so here it is for reference, for those that do not have access:

set -e

PYENV_HOME=$WORKSPACE/.pyenv/
export PIP_DOWNLOAD_CACHE=$WORKSPACE/.pip_cache

# Delete previously built virtualenv
# if [ -d $PYENV_HOME ]; then
    # rm -rf $PYENV_HOME
# fi

# Create virtualenv and install necessary packages
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate

# install mozharness dependencies

pip install --quiet nose
pip install --quiet coverage==3.6b3
pip install --quiet hg-git
pip install --quiet mercurial
pip install --quiet mock
pip install --quiet pep8
pip install --quiet pyflakes
pip install --quiet pylint
pip install --quiet virtualenv

export PYTHONPATH=$WORKSPACE/mozharness:$PYTHONPATH

MOZHARNESS_PY_FILES=`find mozharness/mozharness -name [A-Za-z]\*.py`
SCRIPTS_PY_FILES=`find mozharness/scripts -name [A-Za-z]\*.py`

set +e

# UGLINESS TO GET nosetests --with-xunit working, GAH
HOME=`pwd`/mozharness/test/helper_files coverage run --branch --include 'mozharness/*' $PYENV_HOME/bin/nosetests $(find mozharness/test -type f -name '*.py')
coverage xml -o mozharness/coverage.xml

pyflakes $MOZHARNESS_PY_FILES $SCRIPTS_PY_FILES | tee pyflakes.out
pylint -E -e F -f parseable $MOZHARNESS_PY_FILES $SCRIPTS_PY_FILES | tee pylint.out

# E24 and E226 are part of default ignore in pep8 while E501 is line too long
pep8 mozharness --ignore="E24,E226,E501" | tee pep8.out
exit 0

In the end, which ever way unit tests are invoked, I believe the same set of tests should get run, and we should get equivalent results. Above are three different ways to invoke them: 1) the jenkins script, 2) the unit.sh script, 3) running nosetests directly.

Bug 1073553 talks about a similar issue for the tools repository.

Comment 4

[Pete Moore [:pmoore][:pete]]

So, I pushed, and got a failure, as expected
https://travis-ci.org/mozilla/build-mozharness/builds/41224466:


Using worker: worker-linux-12-2.bb.travis-ci.org:travis-linux-8

system_info

Build system information

Build language: python

git.checkout

0.30s$ git clone --depth=50 --branch=master git://github.com/mozilla/build-mozharness.git mozilla/build-mozharness

Cloning into 'mozilla/build-mozharness'...

remote: Counting objects: 752, done.

remote: Compressing objects: 100% (469/469), done.

remote: Total 752 (delta 418), reused 481 (delta 275)

Receiving objects: 100% (752/752), 509.56 KiB | 0 bytes/s, done.

Resolving deltas: 100% (418/418), done.

Checking connectivity... done.

$ cd mozilla/build-mozharness

$ git checkout -qf a3565dcb0611c47e94837195560c349fdc02579a

0.01s$ source ~/virtualenv/python2.7/bin/activate

$ python --version

Python 2.7.8

$ pip --version

pip 1.5.4 from /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (python 2.7)
install.1

4.68s$ pip install -e .[test]

Obtaining file:///home/travis/build/mozilla/build-mozharness

Running setup.py (path:/home/travis/build/mozilla/build-mozharness/setup.py) egg_info for package from file:///home/travis/build/mozilla/build-mozharness

Installing extra requirements: 'test'

Downloading/unpacking virtualenv (from mozharness==0.7)

Downloading virtualenv-1.11.6-py2.py3-none-any.whl (1.6MB): 1.6MB downloaded

Requirement already satisfied (use --upgrade to upgrade): mock in /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (from mozharness==0.7)

Downloading/unpacking coverage (from mozharness==0.7)

Downloading coverage-3.7.1.tar.gz (284kB): 284kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/coverage/setup.py) egg_info for package coverage

warning: no previously-included files matching '*.pyc' found anywhere in distribution

Requirement already satisfied (use --upgrade to upgrade): nose in /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (from mozharness==0.7)

Downloading/unpacking pylint (from mozharness==0.7)

Downloading pylint-1.3.1.zip (471kB): 471kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/pylint/setup.py) egg_info for package pylint

warning: no files found matching '*.html' under directory 'doc'

Downloading/unpacking pyflakes (from mozharness==0.7)

Downloading pyflakes-0.8.1-py2.py3-none-any.whl

Downloading/unpacking logilab-common>=0.53.0 (from pylint->mozharness==0.7)

Downloading logilab-common-0.63.0.tar.gz (195kB): 195kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/logilab-common/setup.py) egg_info for package logilab-common

package init file './test/__init__.py' not found (or not a regular file)

Downloading/unpacking astroid>=1.2.1 (from pylint->mozharness==0.7)

Downloading astroid-1.2.1.zip (169kB): 169kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/astroid/setup.py) egg_info for package astroid

package init file './test/__init__.py' not found (or not a regular file)

Downloading/unpacking six>=1.4.0 (from logilab-common>=0.53.0->pylint->mozharness==0.7)

Downloading six-1.8.0-py2.py3-none-any.whl

Installing collected packages: virtualenv, coverage, pylint, pyflakes, mozharness, logilab-common, astroid, six

Running setup.py install for coverage

building 'coverage.tracer' extension

gcc -pthread -fno-strict-aliasing -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/python/2.7.8/include/python2.7 -c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o

gcc -pthread -shared -L/opt/python/2.7.8/lib -Wl,-rpath=/opt/python/2.7.8/lib build/temp.linux-x86_64-2.7/coverage/tracer.o -L/opt/python/2.7.8/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/coverage/tracer.so

warning: no previously-included files matching '*.pyc' found anywhere in distribution

Installing coverage2 script to /home/travis/virtualenv/python2.7.8/bin

Installing coverage-2.7 script to /home/travis/virtualenv/python2.7.8/bin

Installing coverage script to /home/travis/virtualenv/python2.7.8/bin

Running setup.py install for pylint

changing mode of build/scripts-2.7/pylint from 664 to 775

changing mode of build/scripts-2.7/pylint-gui from 664 to 775

changing mode of build/scripts-2.7/symilar from 664 to 775

changing mode of build/scripts-2.7/epylint from 664 to 775

changing mode of build/scripts-2.7/pyreverse from 664 to 775

warning: no files found matching '*.html' under directory 'doc'

changing mode of /home/travis/virtualenv/python2.7.8/bin/symilar to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/epylint to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pylint-gui to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pyreverse to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pylint to 775

Installing pylint script to /home/travis/virtualenv/python2.7.8/bin

Installing epylint script to /home/travis/virtualenv/python2.7.8/bin

Installing pyreverse script to /home/travis/virtualenv/python2.7.8/bin

Installing pylint-gui script to /home/travis/virtualenv/python2.7.8/bin

Installing symilar script to /home/travis/virtualenv/python2.7.8/bin

Running setup.py develop for mozharness

Creating /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/mozharness.egg-link (link to .)

Adding mozharness 0.7 to easy-install.pth file

Installed /home/travis/build/mozilla/build-mozharness

Running setup.py install for logilab-common

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

changing mode of build/scripts-2.7/pytest from 664 to 775

package init file './test/__init__.py' not found (or not a regular file)

Installing /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/logilab_common-0.63.0-py2.7-nspkg.pth

changing mode of /home/travis/virtualenv/python2.7.8/bin/pytest to 775

package init file './test/__init__.py' not found (or not a regular file)

Running setup.py install for astroid

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

Successfully installed virtualenv coverage pylint pyflakes mozharness logilab-common astroid six

Cleaning up...
install.2

0.30s$ pip freeze

astroid==1.2.1

coverage==3.7.1

logilab-common==0.63.0

mock==1.0.1

-e git://github.com/mozilla/build-mozharness.git@a3565dcb0611c47e94837195560c349fdc02579a#egg=mozharness-master

nose==1.3.4

numpy==1.9.0

py==1.4.26

pyflakes==0.8.1

pylint==1.3.1

pytest==2.6.4

six==1.8.0

virtualenv==1.11.6

wheel==0.24.0

wsgiref==0.1.2

57.29s$ bash ./run_travis_tests.sh

Running tests...

EEE..................................................S.......S...........................................................................................................

======================================================================

ERROR: Derive and return an appropriate output parser, either the structured

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

TypeError: get_test_output_parser() takes at least 2 arguments (1 given)

======================================================================

ERROR: preflight commands for all tests

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 602, in postflight_run_tests

c = self.config

AttributeError: 'TestingMixin' object has no attribute 'config'

======================================================================

ERROR: preflight commands for all tests

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 590, in preflight_run_tests

self._read_tree_config()

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 374, in _read_tree_config

dirs = self.query_abs_dirs()

AttributeError: 'TestingMixin' object has no attribute 'query_abs_dirs'

----------------------------------------------------------------------

Ran 169 tests in 57.130s

FAILED (SKIP=2, errors=3)

The command "bash ./run_travis_tests.sh" exited with 1.

Done. Your build exited with 1.


However, I did not receive the travis failure email, so need to check who the travis failures get sent to for the mozilla account.

In addition, we should no doubt add pylint and pep8 tests as per jenkins script above in comment 3.

Comment 5

[[e-mail] (:bhearsum)]

Updating summary, because Jenkins may not be the end result.

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_balrog_v1.patch

Results from my fork:
https://travis-ci.org/petemoore/balrog/builds/41251741

Please note I only test for a zero exit code - can you confirm that if any of the steps of the make script fail, that make will return a non-zero exit code?

We also don't have anything to publish failure messages to #releng - we might want to add that too.

Comment 7

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_balrog_v2.patch

Removed redundant line!

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8523874 [details] [diff] [review]
bug1076810_balrog_v2.patch

Review of attachment 8523874 [details] [diff] [review]:
-----------------------------------------------------------------

Yes, notifications would be lovely!

::: run_travis_tests.sh
@@ +2,5 @@
> +
> +set -xveu
> +
> +export PIP_DOWNLOAD_CACHE=.pip_cache
> +make ci-tests "VIRTUALENV_DIR=$(mktemp -d)" "VIRTUALENV=$(which virtualenv)" COVERAGE=1

Rather than adding a new script, why not just update the ci-tests target to do everything we need? That's why it exists :).

Comment 9

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/mozharness/rev/a9abf50b832f

Comment 10

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_balrog_v3.patch

So after a bit of playing around, I realised there were some benefits to using the travis native mechanism for managing the travis ci implementation (.travis.yml file). For example, for each step listed, you get collapsible output, so each step can be expanded/collapsed in the view. If we just call a make script, it would all be in one big output chunk. Also, the ci steps were using a virtualenv, but when travis runs, it has already set up a virtualenv, so then you have the problems of creating a virtualenv when you are already running one. Therefore in the end I decided it was probably simplest to go native-travis style with the .travis.yml where it was clear which steps were being called, and when we decomm jenkins, we could potentially drop the ci-tests target that runs there.

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/4669d1e93c269ab911695c1a690d679c09f5f074
Bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN,r=bhearsum

Comment 12

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_tools_travis-irc_v1.patch

This enables travis to send irc notifications in #releng.

At the moment this will also send notifications for forked repos, but should be a good starting point.

Issue https://github.com/travis-ci/travis-ci/issues/1094 should solve this for the longer term.

Currently the output looks like this:

travis-ci [[e-mail]] entered the room.
travis-ci
15:45:52 petemoore/build-tools#5 (irctest - 6a0a7cd : Pete Moore): The build passed.
15:45:52 Change view : https://github.com/petemoore/build-tools/compare/c9a6eb6d1a1e...6a0a7cd9f48d
15:45:52 Build details : http://travis-ci.org/petemoore/build-tools/builds/41486065
15:45:52 travis-ci left the room.

We can customise this as per:
http://docs.travis-ci.com/user/notifications/#IRC-notification

<snip>

You also have the possibility to customize the message that will be sent to the channel(s) with a template:

notifications:
  irc:
    channels:
      - "chat.freenode.net#my-channel"
      - "chat.freenode.net#some-other-channel"
    template:
      - "%{repository} (%{commit}) : %{message} %{foo} "
      - "Build details: %{build_url}"

You can interpolate the following variables:

    repository_slug: your GitHub repo identifier (like svenfuchs/minimal)
    repository_name: the slug without the username
    repository: same as repository_slug [Deprecated]
    build_number: build number
    build_id: build id
    branch: branch build name
    commit: shortened commit SHA
    author: commit author name
    commit_message: commit message of build
    result: result of build
    message: travis message to the build
    duration: duration of the build
    compare_url: commit change view URL
    build_url: URL of the build detail

</snip>
Pete

Comment 13

[Pete Moore [:pmoore][:pete]]

Callek has also pointed me at https://github.com/lonnen/leeroybot

Comment 14

[Pete Moore [:pmoore][:pete]]

Hey Chris (Lonnen),

How much work would it be to set up leeroybot to work around https://github.com/travis-ci/travis-ci/issues/1094 ?

Basically, we want notifications in #releng channel of all changes/test runs on all branches of the mozilla user for a given repo, but not to get notifications from any user forks.

Would it make sense to piggy-back of an existing service you are running, or set up of our infra for this?

Thanks,
Pete

Comment 15

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_balrog_travis-irc_v1.patch

IRC notifications for balrog

Comment 16

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_travis-irc_v1.patch

IRC notifications for mozharness

Comment 17

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/57d006acf775f5d118737c07231def32a7768114
Bug 1076810 - enable #releng irc notifications from travis-ci for balrog

Comment 18

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8525337 [details] [diff] [review]
bug1076810_balrog_travis-irc_v1.patch

https://github.com/mozilla/balrog/commit/57d006acf775f5d118737c07231def32a7768114

Comment 19

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8525339 [details] [diff] [review]
bug1076810_mozharness_travis-irc_v1.patch

https://hg.mozilla.org/build/mozharness/rev/26784cfb53ee

Comment 20

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/26784cfb53ee

Comment 21

[Lonnen :lonnen]

@pmoore -- we have our bot set up to report pretty specifically, but its easy enough to fork. Easier to work out in IRC when you have some time.

Comment 22

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_fix-nosetests-failures_v1.patch

To fix the three failing tests in mozharness. Essentially, nose thought they were tests, but actually they were methods used by other tests, and not directly tests themselves. If you compare to jenkins results, you will see they were not running in jenkins.

The reason they weren't running in jenkins, is because in the jenkins config, we were specifically scanning two subdirectories of mozharness, so that they would be filtered out. However, I think it is much better to be able to run nosetests out-of-the-box without having to whitelist test directories first, so i simply added the @nottest decorator to these three non-tests, to show nose that they should not be considered as tests, which also fixes the problem. It has the advantage of simplicity - running nosetests is all you have to do, rather than creating a whitelist of subdirs first.

Comment 23

[Massimo Gervasini [:massimo]]

Comment on attachment 8527637 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v1.patch

Thanks Pete!

Comment 24

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8527637 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v1.patch

https://hg.mozilla.org/build/mozharness/rev/9475991241fc

Comment 25

[Pete Moore [:pmoore][:pete]]

(In reply to Chris Lonnen :lonnen from comment #21)
> @pmoore -- we have our bot set up to report pretty specifically, but its
> easy enough to fork. Easier to work out in IRC when you have some time.

Thanks Chris!

I'm 9 hours behind you normally (working in Germany), so our days will not cross, but in Portland all next week - do you fancy meeting up briefly at some point next week to have a quick chat about this?

Pete

Comment 26

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8525324 [details] [diff] [review]
bug1076810_tools_travis-irc_v1.patch

https://hg.mozilla.org/build/tools/rev/e198d932a0b0

Comment 27

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8522990 [details] [diff] [review]
bug1076810_mozharness_v1.patch

https://hg.mozilla.org/build/mozharness/rev/a9abf50b832f

Comment 28

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8527637 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v1.patch

Backed out due to mochitest bustage!

========= Started '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' warnings (results: 1, elapsed: 0 secs) (at 2014-11-24 07:15:26.000165) =========
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain-chunked --total-chunks 5 --this-chunk 1 --blob-upload-branch ash --download-symbols ondemand
 in dir /builds/slave/test/. (timeout 1800 secs) (maxTime 7200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5', '--this-chunk', '1', '--blob-upload-branch', 'ash', '--download-symbols', 'ondemand']
 environment:
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  DISPLAY=:0
  HOME=/home/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MOZ_HIDE_RESULTS_TABLE=1
  MOZ_NODE_PATH=/usr/bin/node
  MOZ_NO_REMOTE=1
  NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
  NO_FAIL_ON_TEST_ERRORS=1
  PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PROPERTIES_FILE=/builds/slave/test/buildprops.json
  PWD=/builds/slave/test
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  XDG_SESSION_COOKIE=5a152710dd62cc268aad214e000002d8-1416841705.484084-12882139
  _=/tools/buildbot/bin/python
 using PTY: False
Traceback (most recent call last):
  File "scripts/scripts/desktop_unittest.py", line 29, in <module>
    from mozharness.mozilla.testing.testbase import TestingMixin, testing_config_options
  File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 15, in <module>
    from nose.tools import nottest
ImportError: No module named nose.tools
program finished with exit code 1
elapsedTime=0.245591
========= Finished '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' warnings (results: 1, elapsed: 0 secs) (at 2014-11-24 07:15:26.780020) =========

Comment 29

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_fix-nosetests-failures_v2.patch

This alternative version works differently: instead of marking the /apparent/ tests as "not a test", which requires the nose libraries which are not used in mochitests, for example, instead we stop the three methods from /looking like/ a test at all, by not having "test" in the name, which means nose never considers them.

This alternative approach should be compatible with nose (so you can still run mozharness tests by issuing the nosetests command) but also does not add new dependencies for desktop_unittest.py or other test classes that inherit from TestingMixin.

This passed on my local branch:
http://travis-ci.org/petemoore/build-mozharness/builds/41984984

Comment 30

[Armen [:armenzg]]

https://hg.mozilla.org/build/ash-mozharness/rev/6395cbb49840

Comment 31

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8527808 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v2.patch

Testing in ash-mozharness with
https://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731

Comment 32

[Armen [:armenzg]]

Comment on attachment 8527808 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v2.patch

Review of attachment 8527808 [details] [diff] [review]:
-----------------------------------------------------------------

jlund, could you please confirm/deny what I say in here?

I know what you're trying to do in here, however, I believe that breaks some assumptions for mozharness.

The prelfight and postflight methods have to match the naming of the action they cover.
http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.py#l1207

I think you need to add this (I'm not sure):
@script.PreScriptAction('run-tests')
@script.PostScriptAction('run-tests')

Comment 33

[Lonnen :lonnen]

@pmoore - sure thing. We'll be sharing a space so I'm sure we'll find some time. On Thursday we'll be doing a day long webdev extravaganza in the larger room and at least then we can do a BOF about setting up a hubot for your team. I'll do a little tutorial too, if you'd like.

Comment 34

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/9475991241fc

Comment 35

[Pete Moore [:pmoore][:pete]]

(In reply to Jordan Lund (:jlund) from comment #34)
> In production: https://hg.mozilla.org/build/mozharness/rev/9475991241fc

The backout was also merged to production, so no overall change:
https://hg.mozilla.org/build/mozharness/rev/6395cbb49840

Comment 36

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_fix-nosetests-failures_v3.patch

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #32)
> Comment on attachment 8527808 [details] [diff] [review]
> bug1076810_mozharness_fix-nosetests-failures_v2.patch
> 
> Review of attachment 8527808 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> jlund, could you please confirm/deny what I say in here?
> 
> I know what you're trying to do in here, however, I believe that breaks some
> assumptions for mozharness.
> 
> The prelfight and postflight methods have to match the naming of the action
> they cover.
> http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.
> py#l1207
> 
> I think you need to add this (I'm not sure):
> @script.PreScriptAction('run-tests')
> @script.PostScriptAction('run-tests')

Thanks Armen, this was a great spot, and I hadn't noticed this requirement, but indeed you are right! Good that I chose you for the review! =)

I've found a maybe simpler way to achieve the same thing:
https://github.com/petemoore/build-mozharness/commit/1f701bdc766055e3ae92bf335192a8b523f3933b

Since this is quite a simple change, and you are sleeping at the moment, I'm send this over to Massimo for review.

Pete

Comment 37

[Massimo Gervasini [:massimo]]

Comment on attachment 8528238 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v3.patch

ship it!

Comment 38

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_fix-nosetests-failures_v4.patch

While I'm at it, I'll get rid of the run_travis_tests.sh file too...

Comment 39

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8528249 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v4.patch

https://hg.mozilla.org/build/mozharness/rev/9795a356ec47

Comment 40

[Armen [:armenzg]]

This produces breakage:
http://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731
https://tbpl.mozilla.org/php/getParsedLog.php?id=53406511&tree=Ash&full=1#error0

You can test it locally by doing this:
hg clone ash-mozharness
cd ash-mozharness
hg up -r ff58508a3731
python scripts/b2g_desktop_unittest.py --cfg b2g/desktop_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 1 --blob-upload-branch ash --download-symbols ondemand --cfg developer_config.py --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tests.zip

Comment 41

[Pete Moore [:pmoore][:pete]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #40)
> This produces breakage:
> http://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731
> https://tbpl.mozilla.org/php/getParsedLog.
> php?id=53406511&tree=Ash&full=1#error0
> 
> You can test it locally by doing this:
> hg clone ash-mozharness
> cd ash-mozharness
> hg up -r ff58508a3731
> python scripts/b2g_desktop_unittest.py --cfg
> b2g/desktop_automation_config.py --test-suite mochitest --this-chunk 1
> --total-chunks 1 --blob-upload-branch ash --download-symbols ondemand --cfg
> developer_config.py --installer-url
> http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-
> linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tar.bz2
> --test-url
> http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-
> linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tests.zip

Hi Armen,

Sorry - I forgot to back that out of Ash Mozharness - will do that now. The final solution was different - this was leftover clean up from experimentation on Ash...

Pete

Comment 42

[Pete Moore [:pmoore][:pete]]

Ah, I see you beat me to it, many thanks!

http://hg.mozilla.org/build/ash-mozharness/rev/dbd4d9fd7627

changeset:   3683:dbd4d9fd7627
tag:         tip
user:        Armen Zambrano Gasparnian <[e-mail]>
date:        Tue Nov 25 15:20:57 2014 -0500
summary:     Backout ff58508a3731

Comment 43

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom-travis-tests_v1.patch

Hi Dustin,

Please note the first time this ran (commit e9e147cfadeaf7fb41f83601934b0d6c5ca2c267), all tests passed:
https://travis-ci.org/petemoore/build-buildbotcustom/builds/42604111

Then I added notifications (commit 96787fd26028b7d6940a0126c3dd037a0d803aaa) and all tests failed:
https://travis-ci.org/petemoore/build-buildbotcustom/builds/42605600

The change was:
$ git diff e9e147cfadeaf7fb41f83601934b0d6c5ca2c267..96787fd26028b7d6940a0126c3dd037a0d803aaa
diff --git a/.travis.yml b/.travis.yml
index 752d7de..8590a3f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -24,7 +24,7 @@ script:
 
 # currently cannot customise per user fork, see:
 # https://github.com/travis-ci/travis-ci/issues/1094
-# notifications:
-#   irc: "irc.mozilla.org#releng"
-#   on_success: always
-#   on_failure: always
+notifications:
+  irc: "irc.mozilla.org#releng"
+  on_success: always
+  on_failure: always
$ 

The notifications are unlikely to have had an effect on the test results, so I assume this is either:

1) An instability in the tests, making them unreliable
2) A problem with the travis imaging process

My guess is number 1 is much more likely.

In any case, I believe these are the correct steps to run the buildbotcustom tests, so I'm going to hope that we can solve the root cause of the above problem independently in a separate bug.

Thanks!
Pete

Comment 44

[Pete Moore [:pmoore][:pete]]

^^^^ Correction: on the second run "22/145 tests errored", not "all tests failed". ^^^^

MaxChangesHandling

testDefaultRepoBigMax ... [ERROR]
testDefaultRepoSmallMax ... [ERROR]
testMergingNoRepoBigMax ... [ERROR]
testMergingNoRepoSmallMax ... [ERROR]
testMergingNoRepoUnlimited ... [ERROR]
testNoRepoBigMax ... [ERROR]
testNoRepoSmallMax ... [ERROR]
testNoRepoUnlimited ... [ERROR]
testRelbranchSmallMax ... [ERROR]

RepoBranchHandling

testDefaultRepoBranch ... [ERROR]
testNoRepoBranch ... [ERROR]
testRelbranch ... [ERROR]

RepositoryIndexParsing

testRepositoryIndexParsing ... [ERROR]

TestPolling

testHgAllLocalesPoller ... [ERROR]
testHgLocalePoller ... [ERROR]
testHgPoller ... [ERROR]

UrlCreation

testOverrideUrl ... [ERROR]
testSimpleUrl ... [ERROR]
testTipsOnlyUrl ... [ERROR]
testTipsOnlyWithLastChangeset ... [ERROR]
testUrlWithLastChangeset ... [ERROR]
testUrlWithUnicodeLastChangeset ... [ERROR]

Comment 45

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8530801 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v1.patch

Review of attachment 8530801 [details] [diff] [review]:
-----------------------------------------------------------------

Regarding the unstable tests .. yeah, that's certainly possible.  The incorrect version of Twisted may have something to do with it.  I don't know anything about the buildbotcustom test suite (if 'hg blame' blames me for even one line, it's long since been forgotten!), so I can't add any additional detail there.

::: .travis.yml
@@ +3,5 @@
> +  - "2.7"
> +
> +install:
> +  - pip install pycrypto
> +  - pip install twisted

I'd really recommend installing the Twisted version used in production here.  Newer Twisteds have deprecated and removed features that older Buildbots used.

I don't think the tests will be sensitive to the versions of any of the other packages here, so the existing pip installs without a version are fine.

@@ +11,5 @@
> +  - pip install pep8
> +  - pip install mock
> +  - pip install pyopenssl
> +  - pip install service-identity
> +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot

https?  And, yuck, full clones.  Is there any way to do the equivalent of --depth 50 here?

@@ +15,5 @@
> +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> +  - cd ~/buildbot/master
> +  - python setup.py install
> +  - git clone --depth 50 --branch=master git://github.com/mozilla/build-tools.git ~/tools
> +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"

weird (though harmless) quoting here..

@@ +19,5 @@
> +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> +
> +script:
> +  - mv "${TRAVIS_BUILD_DIR}" "${TRAVIS_BUILD_DIR%/*}/buildbotcustom" || true
> +  - trial --reporter=verbose $(find "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test" -name '*.py')

The find command shouldn't be necessary here -- just
  trial --reporter=verbose "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test"
should do the trick.

Comment 46

[Pete Moore [:pmoore][:pete]]

Thanks Dustin! Responses below.

(In reply to Dustin J. Mitchell [:dustin] from comment #45)
> Comment on attachment 8530801 [details] [diff] [review]
> bug1076810_buildbotcustom-travis-tests_v1.patch
> 
> Review of attachment 8530801 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Regarding the unstable tests .. yeah, that's certainly possible.  The
> incorrect version of Twisted may have something to do with it.  I don't know
> anything about the buildbotcustom test suite (if 'hg blame' blames me for
> even one line, it's long since been forgotten!), so I can't add any
> additional detail there.
> 
> ::: .travis.yml
> @@ +3,5 @@
> > +  - "2.7"
> > +
> > +install:
> > +  - pip install pycrypto
> > +  - pip install twisted
> 
> I'd really recommend installing the Twisted version used in production here.
> Newer Twisteds have deprecated and removed features that older Buildbots
> used.

Thanks for the tip - I will find out which version we have, and fix it here.

> I don't think the tests will be sensitive to the versions of any of the
> other packages here, so the existing pip installs without a version are fine.
> 
> @@ +11,5 @@
> > +  - pip install pep8
> > +  - pip install mock
> > +  - pip install pyopenssl
> > +  - pip install service-identity
> > +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> 
> https?  And, yuck, full clones.  Is there any way to do the equivalent of
> --depth 50 here?

I will check if I can find a way. Or I can mirror over to github, and then use git clone --depth 50.

> 
> @@ +15,5 @@
> > +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> > +  - cd ~/buildbot/master
> > +  - python setup.py install
> > +  - git clone --depth 50 --branch=master git://github.com/mozilla/build-tools.git ~/tools
> > +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> 
> weird (though harmless) quoting here..

Yes, I guess I can drop the quotes entirely. I kept the ~/ outside the quotes so that it would resolve, but since this is an assignment, technically the quotes are not needed, even if TRAVIS_BUILD_DIR contains any whitespace characters. That is just a personal habit of mine to quote variables that can contain whitespace, but in the case of assignment, word splitting does not occur on the RHS of the assignment.

To demonstrate:

# set example variable potentially containing whitespace
pmoore@Elisandra:~ $ TRAVIS_BUILD_DIR='/home/pmoore/Travis   Builds/builtbotcustom'

# without quotes
pmoore@Elisandra:~ $ echo ~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}
/Users/pmoore/tools/lib/python:/home/pmoore/Travis Builds

# with quotes around everything
pmoore@Elisandra:~ $ echo "~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
~/tools/lib/python:/home/pmoore/Travis   Builds

# with quotes around everything but not the ~/
pmoore@Elisandra:~ $ echo ~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
/Users/pmoore/tools/lib/python:/Users/pmoore/Travis   Builds

However, the 'echo' case is different to the assignment case, as the shell does not perform word splitting on the RHS of an assignment. So:

# without quotes - works
pmoore@Elisandra:~ $ PYTHONPATH=~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}; echo "PYTHONPATH: '${PYTHONPATH}'"
PYTHONPATH: '/Users/pmoore/tools/lib/python:/home/pmoore/Travis   Builds'

# with quotes - also works
pmoore@Elisandra:~ $ PYTHONPATH=~"/tools/lib/python:${TRAVIS_BUILD_DIR%/*}"; echo "PYTHONPATH: '${PYTHONPATH}'"
PYTHONPATH: '~/tools/lib/python:/home/pmoore/Travis   Builds'


Therefore I'll ditch the quotes! Sorry for the long explanation!


> 
> @@ +19,5 @@
> > +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> > +
> > +script:
> > +  - mv "${TRAVIS_BUILD_DIR}" "${TRAVIS_BUILD_DIR%/*}/buildbotcustom" || true
> > +  - trial --reporter=verbose $(find "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test" -name '*.py')
> 
> The find command shouldn't be necessary here -- just
>   trial --reporter=verbose "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test"
> should do the trick.

There was no __init__.py in the buildbotcustom/test directory so passing the directory name did not work but I can add __init__.py add at the same time I'll simplify this to:
trial --reporter=verbose buildbotcustom.test

Thanks!
Pete

Comment 47

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom-travis-tests_v2.patch

Comment 48

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_tools_travis-irc_v2.patch

Added verbose mode to nosetests (-v).

Comment 49

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8532342 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v2.patch

https://hg.mozilla.org/build/buildbotcustom/rev/4beb2f22fb96

Comment 50

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8532346 [details] [diff] [review]
bug1076810_tools_travis-irc_v2.patch

https://hg.mozilla.org/build/tools/rev/147d97b51481

Comment 51

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8532342 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v2.patch

Missing execute permissions, fixed:
https://hg.mozilla.org/build/buildbotcustom/rev/8d4acc31bdf5

Comment 52

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/4beb2f22fb96

Comment 53

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/8d4acc31bdf5

Comment 54

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/f753e0ec69b9

Comment 55

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom-travis-tests_v3.patch

This patch is in addition to previous ones (not a replacement).

It:

  * enables coverage reporting, and publishing to coveralls: https://coveralls.io/r/mozilla/build-buildbotcustom
  * enables tox to work locally, even if your checkout is not in a dir buildbotcustom (e.g. build-buildbotcustom because you clone from github)
  * adds a README.md file for presenting in github, explaining how to run tests locally
  * updates .hgignore and .gitignore to exclude test reports and coverage reports

Comment 56

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8532693 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v3.patch

https://hg.mozilla.org/build/buildbotcustom/rev/13053f5ca356

Comment 57

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom-travis-tests_v4.patch

Improve README.md file for github viewers, and include commit message in travis notifications.

Comment 58

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8533176 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v4.patch

r+'d by rail over IRC

landed:
https://hg.mozilla.org/build/buildbotcustom/rev/2e7b5a881893

Comment 59

[Jordan Lund (:jlund)]

Attachment: 141208_bug_1076810-add_python_binaries_to_gitignore-bbotcustom.patch

I had a gitignore before in my working repo. adding the python bits that I prefer to ignore so my 'git status' stays clean.

Comment 60

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8533256 [details] [diff] [review]
141208_bug_1076810-add_python_binaries_to_gitignore-bbotcustom.patch

Review of attachment 8533256 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks Jordan!

Comment 61

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8533176 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v4.patch

https://hg.mozilla.org/build/buildbotcustom/rev/2e7b5a881893

Comment 62

[Jordan Lund (:jlund)]

Comment on attachment 8533256 [details] [diff] [review]
141208_bug_1076810-add_python_binaries_to_gitignore-bbotcustom.patch

default: https://hg.mozilla.org/build/buildbotcustom/rev/5bcd7430ed32

Comment 63

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/13053f5ca356

Comment 64

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/2e7b5a881893

Comment 65

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/5bcd7430ed32

Comment 66

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_tox-coveralls-readme-coloured-verbose-tests_v1.patch

Hey Massimo,

This patch provides the following changes:
1) converts mozharness testing over to tox, to align with our other repos
2) a bug fix for the issue whereby no command line options could be used when running nosetests
3) switches nosetests to used verbose and colourised logging
4) adds coverage reports, and publishes them to http://coveralls.io/
5) adds a README.md to the project, so that when viewed in github, from the front page users have an overview of the project, and useful links to find out more information, and instructions on how to test it, and run tests in travis/coveralls.

Comment 67

[Pete Moore [:pmoore][:pete]]

... and travis results here: https://travis-ci.org/petemoore/build-mozharness/builds/44570978

Comment 68

[Massimo Gervasini [:massimo]]

Comment on attachment 8539236 [details] [diff] [review]
bug1076810_mozharness_tox-coveralls-readme-coloured-verbose-tests_v1.patch

Thanks Pete!

Comment 69

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8539236 [details] [diff] [review]
bug1076810_mozharness_tox-coveralls-readme-coloured-verbose-tests_v1.patch

https://hg.mozilla.org/build/mozharness/rev/e110bc8e37b4

Comment 70

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_tools_coveralls-readme-coloured-output_v1.patch

Hey Rail,

This patch:
1) Colourises the test output, making it easier to see at a glance which tests passed/failed
2) Publishes the coverage reports to coveralls (and does not cause build to fail if publish fails)
3) Adds a README.md to tools, for getting an overview when viewing in github

Comment 71

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8539440 [details] [diff] [review]
bug1076810_tools_coveralls-readme-coloured-output_v1.patch

https://hg.mozilla.org/build/tools/rev/ffe43382c02d

Comment 72

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom_coveralls-publish-not-to-cause-failures_v1.patch

We've just had a couple of failures that were actually just coveralls publishing problems. This shouldn't cause the test result to be failure, so I've moved this to "after_script" section which will not impact job result.

Comment 73

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_coveralls-publish-not-to-cause-failures_v1.patch

... and same for mozharness

Comment 74

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8539454 [details] [diff] [review]
bug1076810_mozharness_coveralls-publish-not-to-cause-failures_v1.patch

https://hg.mozilla.org/build/mozharness/rev/701d2eda2aec

Comment 75

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8539453 [details] [diff] [review]
bug1076810_buildbotcustom_coveralls-publish-not-to-cause-failures_v1.patch

https://hg.mozilla.org/build/buildbotcustom/rev/23df141cf12d

Comment 76

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/23df141cf12d

Comment 77

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/mozharness/rev/701d2eda2aec

Comment 78

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_readme-improvements_v1.patch

Added information to README.md about matching git SHAs to hg SHAs and also that we don't support github pull request workflow, with a bit more detail about process.

Comment 79

[Massimo Gervasini [:massimo]]

Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

Thanks Pete!

Comment 80

[Massimo Gervasini [:massimo]]

Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

Sorry, it was a r+ not a checked-in

Comment 81

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

https://hg.mozilla.org/build/mozharness/rev/5b95f2ee37ff

Comment 82

[Jordan Lund (:jlund)]

mozharness has been merged to production. patches are live :)

Comment 83

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbot-configs_v1.patch

Hey Callek,

Here is my work in progress! We are hitting the pip install bug, but it sounds like you have a workaround for this, so here is my work to date. Should we land this, and then apply a separate patch for your pip fix? Or should we roll it all up into one patch?

Pete

Comment 84

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbot-configs_v2.patch

r+'d by Callek with some changes (will attach separately)

Comment 85

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbot-configs_pep8-failures-allowed_v1.patch

This is Callek's patch, which I am reviewing.

Comment 86

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8544474 [details] [diff] [review]
bug1076810_buildbot-configs_pep8-failures-allowed_v1.patch

https://hg.mozilla.org/build/buildbot-configs/rev/a599d98c55de

Comment 87

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8544473 [details] [diff] [review]
bug1076810_buildbot-configs_v2.patch

https://hg.mozilla.org/build/buildbot-configs/rev/222b655abc77

Comment 88

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_buildbotcustom_fix-irc-notifications-to-include-commit-message_v1.patch

Here is a travis irc notification from earlier today, which demonstrates the commit message missing:

travis-ci [[e-mail]] entered the room.
travis-ci
05:38:25 mozilla/build-buildbotcustom#51 (production-0.8 - 4946124 : ffxbld): The build passed.
05:38:25 Change view : https://github.com/mozilla/build-buildbotcustom/compare/ae779e11de0c...49461241bf85
05:38:25 Build details : http://travis-ci.org/mozilla/build-buildbotcustom/builds/46032679
05:38:25 travis-ci left the room.

With this patch applied (in my personal fork) here the commit message is now included:

travis-ci [[e-mail]] entered the room.
travis-ci
10:23:32 petemoore/build-buildbotcustom#56 (bug1076810 - 64e3406 : Pete Moore): The build passed.
10:23:32 Change view : https://github.com/petemoore/build-buildbotcustom/compare/5ae3bb94626f...64e3406662c3
10:23:32 Build details : http://travis-ci.org/petemoore/build-buildbotcustom/builds/46048372
10:23:32 Commit message : Include commit message in IRC notification.
10:23:32 travis-ci left the room.

Thanks Massimo!
Pete

Comment 89

[Massimo Gervasini [:massimo]]

Comment on attachment 8544502 [details] [diff] [review]
bug1076810_buildbotcustom_fix-irc-notifications-to-include-commit-message_v1.patch

Thanks Pete!

Comment 90

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8544502 [details] [diff] [review]
bug1076810_buildbotcustom_fix-irc-notifications-to-include-commit-message_v1.patch

https://hg.mozilla.org/build/buildbotcustom/rev/4a94dc845f16

Comment 91

[Pete Moore [:pmoore][:pete]]

Attachment: Adding irc notifications to balrog-ui repo

This repo had failing builds, so I thought it would be good to add irc notifications so that it is more visible when it starts failing.

Thanks Massimo!
Pete

Comment 92

[Massimo Gervasini [:massimo]]

Comment on attachment 8544540 [details] [review]
Adding irc notifications to balrog-ui repo

https://github.com/mozilla/balrog-ui/pull/6/files looks fine, thanks Pete.

Comment 93

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8544540 [details] [review]
Adding irc notifications to balrog-ui repo

https://github.com/mozilla/balrog-ui/commit/88bbb95798fa2aea2b6c4f582378d5bc7de159e0

Comment 94

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/222b655abc77

Comment 95

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/a599d98c55de

Comment 96

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/4a94dc845f16

Comment 97

[Pete Moore [:pmoore][:pete]]

Some important notes, as I am not giving this bug much attention at the moment, if anyone wants to pick up parts of it...

STILL TO DO
===========
It is worth highlighting here, that in our jenkins setup, if any of the following change:
  * buildbot-configs
  * tools
  * buildbotcustom
then we would run:
  buildbot-configs/test-masters.sh -e
Currently this is only triggered in travis for buildbot-configs changes. This means this test should also be added to the travis jobs for tools and buildbotcustom.


IRC NOTIFICATIONS
=================

clonnen also provided very useful info about how to filter irc messages using lonnenbot:
https://gist.github.com/lonnen/8699bf472c98ba34e580

I haven't done this part either yet.

As per comment 12, lonnenbot is not not needed if/when https://github.com/travis-ci/travis-ci/issues/1094 is resolved. However, if it is resolved, we should then audit all of our .travis.yml files and make sure we report consistently across our repos, e.g. that commit_message is always present in the irc notification messages, like it is here: https://github.com/mozilla/build-buildbot-configs/blob/3e8d3330620b2c7ed463372a033a6fbf650ee072/.travis.yml#L24


JORDAN'S COOL IDEAS
===================

Jordan (jlund) also noted in email:

 Then, add builder_list.py and dump_master.py runs on top of test-masters.sh. Finally, serve the results of those tests to s3 or somewhere internally within travis.

Whenever I get a buildbot r?, I run the patch with: https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_patch.sh
    e.g. $ ~/bin/dotfiles/mozilla/verify_patch.sh -e test -f $DOW/reftestchunks.patch  # apply reftestchunks.patch to bbot-cfgs repo + and get a clean/dirty copy of builder_list and dump_master diff against uni test master

If I have my own dev branch, I run the following before submitting a patch: https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_branch.sh
    e.g. $ /bin/dotfiles/mozilla/verify_branch.sh -e test -u clone_less -f clone_less  # similar to above but compare master <-> clone_less branch and use buildbot-config (-f) and buildbotcustom (-u)

we could customize scripts like these to suit our travis setup.

We shouldn't lose track of these cool ideas, so adding this to this bug, as I'm not sure how much more I'll be working on this.


TRAVIS DASHBOARD
================

Jenkins allowed us to have a simple page view of all jobs, to see which were ok/running/failing tests/erroring:
http://10.134.48.37:8080/

(access to Mozilla network required)

We miss this with travis. Something like this gives a close approximation:
http://egeloen.fr/travis-wall/#/petemoore

But it would be worth spending a little time to create a dashboard that has no personal results, and has all the RelEng repos on it, unlike the one above. Probably not a lot of work, but would help enormously, and should probably be integrated somewhere prominent in RelEng world.


SWITCHING OFF JENKINS
=====================

Probably when all of the above is done (with the possible exception of the enhancement ideas of Jordan) we can finally switch of Jenkins.

Comment 98

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_mozharness_customise-irc-output_v1.patch

Adding commit message to mozharness irc notifications (to be consistent with other projects)...

Comment 99

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8545823 [details] [diff] [review]
bug1076810_mozharness_customise-irc-output_v1.patch

https://hg.mozilla.org/build/mozharness/rev/f8b44474c06e

Comment 100

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/f8b44474c06e

Comment 101

[Pete Moore [:pmoore][:pete]]

Attachment: bug1076810_tools_travis-irc_v3.patch

I spotted that tools irc messages missed the commit message:

e.g.

travis-ci
15:08:35 mozilla/build-tools#273 (master - e46c4bf : ffxbld): The build passed.
15:08:35 Change view : https://github.com/mozilla/build-tools/compare/8e1d9d06fdf3...e46c4bf47ca5
15:08:35 Build details : http://travis-ci.org/mozilla/build-tools/builds/53191909

Compare this to e.g. mozharness:
 mozilla/build-mozharness#321 (production - e4ea4fd : ffxbld): The build passed.
15:09:11 Change view : https://github.com/mozilla/build-mozharness/compare/45845c8fb610...e4ea4fd22659
15:09:11 Build details : http://travis-ci.org/mozilla/build-mozharness/builds/53193288
15:09:11 Commit message : Added FENNEC_36_0_1_RELEASE FENNEC_36_0_1_BUILD2 tag(s) for changeset production. DONTBUILD CLOSED TREE a=release


This patch adds the extra "Commit message" line.

Comment 102

[Massimo Gervasini [:massimo]]

Comment on attachment 8573256 [details] [diff] [review]
bug1076810_tools_travis-irc_v3.patch

Thanks Pete!

Comment 103

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8573256 [details] [diff] [review]
bug1076810_tools_travis-irc_v3.patch

https://hg.mozilla.org/build/tools/rev/f3ff5189ca46

Comment 104

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pete][:pmoore] from comment #97)
> Some important notes, as I am not giving this bug much attention at the
> moment, if anyone wants to pick up parts of it...
> 
> STILL TO DO
> ===========
> It is worth highlighting here, that in our jenkins setup, if any of the
> following change:
>   * buildbot-configs
>   * tools
>   * buildbotcustom
> then we would run:
>   buildbot-configs/test-masters.sh -e
> Currently this is only triggered in travis for buildbot-configs changes.
> This means this test should also be added to the travis jobs for tools and
> buildbotcustom.
> 

The outstanding bits related to buildbot become not so important in the light of moving jobs to Task Cluster. 

> 
> IRC NOTIFICATIONS
> =================
> 
> clonnen also provided very useful info about how to filter irc messages
> using lonnenbot:
> https://gist.github.com/lonnen/8699bf472c98ba34e580
> 
> I haven't done this part either yet.
> 
> As per comment 12, lonnenbot is not not needed if/when
> https://github.com/travis-ci/travis-ci/issues/1094 is resolved. However, if
> it is resolved, we should then audit all of our .travis.yml files and make
> sure we report consistently across our repos, e.g. that commit_message is
> always present in the irc notification messages, like it is here:
> https://github.com/mozilla/build-buildbot-configs/blob/
> 3e8d3330620b2c7ed463372a033a6fbf650ee072/.travis.yml#L24
> 

Since travis now have it on the roadmap to fix issue 1094, I think this isn't worth the investment now for a temporary solution, to an issue that is not so important.

> 
> JORDAN'S COOL IDEAS
> ===================
> 
> Jordan (jlund) also noted in email:
> 
>  Then, add builder_list.py and dump_master.py runs on top of
> test-masters.sh. Finally, serve the results of those tests to s3 or
> somewhere internally within travis.
> 
> Whenever I get a buildbot r?, I run the patch with:
> https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_patch.sh
>     e.g. $ ~/bin/dotfiles/mozilla/verify_patch.sh -e test -f
> $DOW/reftestchunks.patch  # apply reftestchunks.patch to bbot-cfgs repo +
> and get a clean/dirty copy of builder_list and dump_master diff against uni
> test master
> 
> If I have my own dev branch, I run the following before submitting a patch:
> https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_branch.sh
>     e.g. $ /bin/dotfiles/mozilla/verify_branch.sh -e test -u clone_less -f
> clone_less  # similar to above but compare master <-> clone_less branch and
> use buildbot-config (-f) and buildbotcustom (-u)
> 
> we could customize scripts like these to suit our travis setup.
> 
> We shouldn't lose track of these cool ideas, so adding this to this bug, as
> I'm not sure how much more I'll be working on this.
> 

This is all buildbot related, so move to Task Cluster negates the need to implement this now.

> 
> TRAVIS DASHBOARD
> ================
> 
> Jenkins allowed us to have a simple page view of all jobs, to see which were
> ok/running/failing tests/erroring:
> http://10.134.48.37:8080/
> 
> (access to Mozilla network required)
> 
> We miss this with travis. Something like this gives a close approximation:
> http://egeloen.fr/travis-wall/#/petemoore
> 
> But it would be worth spending a little time to create a dashboard that has
> no personal results, and has all the RelEng repos on it, unlike the one
> above. Probably not a lot of work, but would help enormously, and should
> probably be integrated somewhere prominent in RelEng world.
> 

Callek has kindly created:
https://wiki.mozilla.org/ReleaseEngineering/Repositories

> 
> SWITCHING OFF JENKINS
> =====================
> 
> Probably when all of the above is done (with the possible exception of the
> enhancement ideas of Jordan) we can finally switch of Jenkins.

Checking now with catlee/bhearsum if we can do this...

Based on all of the above, closing this bug now.

Comment 105

[[github robot]]

Commits pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/cd695e6f30c3047c3cb4f41d5141fbcb2cea0071
[balrog-ui] Bug 1076810 - RelEng CI tests should be available over the internet, add irc notifications for balrog-ui repository,r=mgerva

https://github.com/mozilla/balrog/commit/874e9f122483e0a0bb6747f5515b380280f4001e
[balrog-ui] Merge pull request #6 from petemoore/bug1076810

Bug 1076810 - RelEng CI tests should be available over the internet, add irc notifications for balrog-ui repository,r=mgerva