Closed Bug 1076810 Opened 10 years ago Closed 9 years ago

RelEng CI tests should be available over the internet, not just from VPN

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: pmoore, Assigned: pmoore)

References

()

Details

Attachments

(22 files, 7 obsolete files)

479 bytes, patch
armenzg
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
910 bytes, patch
bhearsum
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
354 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
477 bytes, patch
bhearsum
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
373 bytes, patch
armenzg
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
771 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
11.91 KB, patch
rail
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
413 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
4.20 KB, patch
rail
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
1.36 KB, patch
pmoore
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
159 bytes, patch
pmoore
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
3.78 KB, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
2.17 KB, patch
rail
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
472 bytes, patch
rail
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
457 bytes, patch
rail
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
3.19 KB, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
6.00 KB, patch
pmoore
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
1.69 KB, patch
pmoore
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
951 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
43 bytes, text/x-github-pull-request
massimo
: review+
pmoore
: checked-in+
Details | Review
730 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
758 bytes, patch
massimo
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
Currently jenkins is hosted on an internal private IP (http://10.134.48.37:8080/). AFAIK there is no natting to expose this publicly.

This means contributors cannot see our CI results, which is a big reason not to contribute!

An alternative to exposing this service publicly, would be to migrate over to travis, for example, which also might save some costs, but also come at some cost to migrate.

I'm also aware other teams than RelEng (such as A Team) rely heavily on mozharness, so they may wish to enter into the discussion of where they would like mozharness CI to live - and of course this is also CI for other repos, like buildbot, tools.

If we do decide to expose it, maybe it should move over to Dev Services management?

If we move to travis, please note there may be a slight further delay (~1 minute) until results are published, because our github repos lag around 1 minute behind our hg repos, due to the time it takes to mirror changes from hg.m.o -> github. However, it considerably increases visibility.
+1!

I don't find these results critical to get them fast, however, it is nice to make them public.
So, out of the box the mozharness tests are not working, but I don't think that should stop the delivery of standing up the CI in travis - rather I'd like to get travis CI enabled for the various build repos, and then run through the issues to fix them.

Check out https://travis-ci.org/petemoore/build-mozharness to see results.
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #8522990 - Flags: review?(armenzg)
Attachment #8522990 - Flags: review?(armenzg) → review+
I've spotted we have a script in mozharness for running unit tests (https://github.com/mozilla/build-mozharness/blob/master/unit.sh), but I think I still prefer calling nosetests directly that running this script because:

1) the unit test script is also running coverage tests, and afaik, we have nowhere to publish the reports when running from travis, so this could be wasted effort
2) I think it is a fair requirement that the nosetests command should run all the unit tests correctly out-of-the-box, so we shouldn't need a bash script to invoke it which adds other logic too - based on this I'd like to land the current patch, and then work on fixing up the tests so that:
a) all of them run that can run,
b) they all pass

There are also a third set of test results, which we currently get in our jenkins CI (which travis would replace). I've managed to get admin access to the job configuration, so here it is for reference, for those that do not have access:

set -e

PYENV_HOME=$WORKSPACE/.pyenv/
export PIP_DOWNLOAD_CACHE=$WORKSPACE/.pip_cache

# Delete previously built virtualenv
# if [ -d $PYENV_HOME ]; then
    # rm -rf $PYENV_HOME
# fi

# Create virtualenv and install necessary packages
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate

# install mozharness dependencies

pip install --quiet nose
pip install --quiet coverage==3.6b3
pip install --quiet hg-git
pip install --quiet mercurial
pip install --quiet mock
pip install --quiet pep8
pip install --quiet pyflakes
pip install --quiet pylint
pip install --quiet virtualenv

export PYTHONPATH=$WORKSPACE/mozharness:$PYTHONPATH

MOZHARNESS_PY_FILES=`find mozharness/mozharness -name [A-Za-z]\*.py`
SCRIPTS_PY_FILES=`find mozharness/scripts -name [A-Za-z]\*.py`

set +e

# UGLINESS TO GET nosetests --with-xunit working, GAH
HOME=`pwd`/mozharness/test/helper_files coverage run --branch --include 'mozharness/*' $PYENV_HOME/bin/nosetests $(find mozharness/test -type f -name '*.py')
coverage xml -o mozharness/coverage.xml

pyflakes $MOZHARNESS_PY_FILES $SCRIPTS_PY_FILES | tee pyflakes.out
pylint -E -e F -f parseable $MOZHARNESS_PY_FILES $SCRIPTS_PY_FILES | tee pylint.out

# E24 and E226 are part of default ignore in pep8 while E501 is line too long
pep8 mozharness --ignore="E24,E226,E501" | tee pep8.out
exit 0

In the end, which ever way unit tests are invoked, I believe the same set of tests should get run, and we should get equivalent results. Above are three different ways to invoke them: 1) the jenkins script, 2) the unit.sh script, 3) running nosetests directly.

Bug 1073553 talks about a similar issue for the tools repository.
So, I pushed, and got a failure, as expected
https://travis-ci.org/mozilla/build-mozharness/builds/41224466:


Using worker: worker-linux-12-2.bb.travis-ci.org:travis-linux-8

system_info

Build system information

Build language: python

git.checkout

0.30s$ git clone --depth=50 --branch=master git://github.com/mozilla/build-mozharness.git mozilla/build-mozharness

Cloning into 'mozilla/build-mozharness'...

remote: Counting objects: 752, done.

remote: Compressing objects: 100% (469/469), done.

remote: Total 752 (delta 418), reused 481 (delta 275)

Receiving objects: 100% (752/752), 509.56 KiB | 0 bytes/s, done.

Resolving deltas: 100% (418/418), done.

Checking connectivity... done.

$ cd mozilla/build-mozharness

$ git checkout -qf a3565dcb0611c47e94837195560c349fdc02579a

0.01s$ source ~/virtualenv/python2.7/bin/activate

$ python --version

Python 2.7.8

$ pip --version

pip 1.5.4 from /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (python 2.7)
install.1

4.68s$ pip install -e .[test]

Obtaining file:///home/travis/build/mozilla/build-mozharness

Running setup.py (path:/home/travis/build/mozilla/build-mozharness/setup.py) egg_info for package from file:///home/travis/build/mozilla/build-mozharness

Installing extra requirements: 'test'

Downloading/unpacking virtualenv (from mozharness==0.7)

Downloading virtualenv-1.11.6-py2.py3-none-any.whl (1.6MB): 1.6MB downloaded

Requirement already satisfied (use --upgrade to upgrade): mock in /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (from mozharness==0.7)

Downloading/unpacking coverage (from mozharness==0.7)

Downloading coverage-3.7.1.tar.gz (284kB): 284kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/coverage/setup.py) egg_info for package coverage

warning: no previously-included files matching '*.pyc' found anywhere in distribution

Requirement already satisfied (use --upgrade to upgrade): nose in /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages (from mozharness==0.7)

Downloading/unpacking pylint (from mozharness==0.7)

Downloading pylint-1.3.1.zip (471kB): 471kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/pylint/setup.py) egg_info for package pylint

warning: no files found matching '*.html' under directory 'doc'

Downloading/unpacking pyflakes (from mozharness==0.7)

Downloading pyflakes-0.8.1-py2.py3-none-any.whl

Downloading/unpacking logilab-common>=0.53.0 (from pylint->mozharness==0.7)

Downloading logilab-common-0.63.0.tar.gz (195kB): 195kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/logilab-common/setup.py) egg_info for package logilab-common

package init file './test/__init__.py' not found (or not a regular file)

Downloading/unpacking astroid>=1.2.1 (from pylint->mozharness==0.7)

Downloading astroid-1.2.1.zip (169kB): 169kB downloaded

Running setup.py (path:/home/travis/virtualenv/python2.7.8/build/astroid/setup.py) egg_info for package astroid

package init file './test/__init__.py' not found (or not a regular file)

Downloading/unpacking six>=1.4.0 (from logilab-common>=0.53.0->pylint->mozharness==0.7)

Downloading six-1.8.0-py2.py3-none-any.whl

Installing collected packages: virtualenv, coverage, pylint, pyflakes, mozharness, logilab-common, astroid, six

Running setup.py install for coverage

building 'coverage.tracer' extension

gcc -pthread -fno-strict-aliasing -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/python/2.7.8/include/python2.7 -c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o

gcc -pthread -shared -L/opt/python/2.7.8/lib -Wl,-rpath=/opt/python/2.7.8/lib build/temp.linux-x86_64-2.7/coverage/tracer.o -L/opt/python/2.7.8/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/coverage/tracer.so

warning: no previously-included files matching '*.pyc' found anywhere in distribution

Installing coverage2 script to /home/travis/virtualenv/python2.7.8/bin

Installing coverage-2.7 script to /home/travis/virtualenv/python2.7.8/bin

Installing coverage script to /home/travis/virtualenv/python2.7.8/bin

Running setup.py install for pylint

changing mode of build/scripts-2.7/pylint from 664 to 775

changing mode of build/scripts-2.7/pylint-gui from 664 to 775

changing mode of build/scripts-2.7/symilar from 664 to 775

changing mode of build/scripts-2.7/epylint from 664 to 775

changing mode of build/scripts-2.7/pyreverse from 664 to 775

warning: no files found matching '*.html' under directory 'doc'

changing mode of /home/travis/virtualenv/python2.7.8/bin/symilar to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/epylint to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pylint-gui to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pyreverse to 775

changing mode of /home/travis/virtualenv/python2.7.8/bin/pylint to 775

Installing pylint script to /home/travis/virtualenv/python2.7.8/bin

Installing epylint script to /home/travis/virtualenv/python2.7.8/bin

Installing pyreverse script to /home/travis/virtualenv/python2.7.8/bin

Installing pylint-gui script to /home/travis/virtualenv/python2.7.8/bin

Installing symilar script to /home/travis/virtualenv/python2.7.8/bin

Running setup.py develop for mozharness

Creating /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/mozharness.egg-link (link to .)

Adding mozharness 0.7 to easy-install.pth file

Installed /home/travis/build/mozilla/build-mozharness

Running setup.py install for logilab-common

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

changing mode of build/scripts-2.7/pytest from 664 to 775

package init file './test/__init__.py' not found (or not a regular file)

Installing /home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/logilab_common-0.63.0-py2.7-nspkg.pth

changing mode of /home/travis/virtualenv/python2.7.8/bin/pytest to 775

package init file './test/__init__.py' not found (or not a regular file)

Running setup.py install for astroid

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

package init file './test/__init__.py' not found (or not a regular file)

Successfully installed virtualenv coverage pylint pyflakes mozharness logilab-common astroid six

Cleaning up...
install.2

0.30s$ pip freeze

astroid==1.2.1

coverage==3.7.1

logilab-common==0.63.0

mock==1.0.1

-e git://github.com/mozilla/build-mozharness.git@a3565dcb0611c47e94837195560c349fdc02579a#egg=mozharness-master

nose==1.3.4

numpy==1.9.0

py==1.4.26

pyflakes==0.8.1

pylint==1.3.1

pytest==2.6.4

six==1.8.0

virtualenv==1.11.6

wheel==0.24.0

wsgiref==0.1.2

57.29s$ bash ./run_travis_tests.sh

Running tests...

EEE..................................................S.......S...........................................................................................................

======================================================================

ERROR: Derive and return an appropriate output parser, either the structured

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

TypeError: get_test_output_parser() takes at least 2 arguments (1 given)

======================================================================

ERROR: preflight commands for all tests

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 602, in postflight_run_tests

c = self.config

AttributeError: 'TestingMixin' object has no attribute 'config'

======================================================================

ERROR: preflight commands for all tests

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

self.test(*self.arg)

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 590, in preflight_run_tests

self._read_tree_config()

File "/home/travis/build/mozilla/build-mozharness/mozharness/mozilla/testing/testbase.py", line 374, in _read_tree_config

dirs = self.query_abs_dirs()

AttributeError: 'TestingMixin' object has no attribute 'query_abs_dirs'

----------------------------------------------------------------------

Ran 169 tests in 57.130s

FAILED (SKIP=2, errors=3)

The command "bash ./run_travis_tests.sh" exited with 1.

Done. Your build exited with 1.


However, I did not receive the travis failure email, so need to check who the travis failures get sent to for the mozilla account.

In addition, we should no doubt add pylint and pep8 tests as per jenkins script above in comment 3.
See Also: → 1073553
Updating summary, because Jenkins may not be the end result.
Summary: RelEng Jenkins tests should be available over the internet, not just from VPN → RelEng CI tests should be available over the internet, not just from VPN
Depends on: 1073553
See Also: 1073553
Attached patch bug1076810_balrog_v1.patch (obsolete) — Splinter Review
Results from my fork:
https://travis-ci.org/petemoore/balrog/builds/41251741

Please note I only test for a zero exit code - can you confirm that if any of the steps of the make script fail, that make will return a non-zero exit code?

We also don't have anything to publish failure messages to #releng - we might want to add that too.
Attachment #8523871 - Flags: review?(bhearsum)
Attached patch bug1076810_balrog_v2.patch (obsolete) — Splinter Review
Removed redundant line!
Attachment #8523871 - Attachment is obsolete: true
Attachment #8523871 - Flags: review?(bhearsum)
Attachment #8523874 - Flags: review?(bhearsum)
Comment on attachment 8523874 [details] [diff] [review]
bug1076810_balrog_v2.patch

Review of attachment 8523874 [details] [diff] [review]:
-----------------------------------------------------------------

Yes, notifications would be lovely!

::: run_travis_tests.sh
@@ +2,5 @@
> +
> +set -xveu
> +
> +export PIP_DOWNLOAD_CACHE=.pip_cache
> +make ci-tests "VIRTUALENV_DIR=$(mktemp -d)" "VIRTUALENV=$(which virtualenv)" COVERAGE=1

Rather than adding a new script, why not just update the ci-tests target to do everything we need? That's why it exists :).
So after a bit of playing around, I realised there were some benefits to using the travis native mechanism for managing the travis ci implementation (.travis.yml file). For example, for each step listed, you get collapsible output, so each step can be expanded/collapsed in the view. If we just call a make script, it would all be in one big output chunk. Also, the ci steps were using a virtualenv, but when travis runs, it has already set up a virtualenv, so then you have the problems of creating a virtualenv when you are already running one. Therefore in the end I decided it was probably simplest to go native-travis style with the .travis.yml where it was clear which steps were being called, and when we decomm jenkins, we could potentially drop the ci-tests target that runs there.
Attachment #8523874 - Attachment is obsolete: true
Attachment #8523874 - Flags: review?(bhearsum)
Attachment #8524474 - Flags: review?(bhearsum)
Attachment #8524474 - Flags: review?(bhearsum) → review+
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/4669d1e93c269ab911695c1a690d679c09f5f074
Bug 1076810 - RelEng CI tests should be available over the internet, not just from VPN,r=bhearsum
Attachment #8524474 - Flags: checked-in+
This enables travis to send irc notifications in #releng.

At the moment this will also send notifications for forked repos, but should be a good starting point.

Issue https://github.com/travis-ci/travis-ci/issues/1094 should solve this for the longer term.

Currently the output looks like this:

travis-ci [travis-ci@moz-21ggri.compute-1.amazonaws.com] entered the room.
travis-ci
15:45:52 petemoore/build-tools#5 (irctest - 6a0a7cd : Pete Moore): The build passed.
15:45:52 Change view : https://github.com/petemoore/build-tools/compare/c9a6eb6d1a1e...6a0a7cd9f48d
15:45:52 Build details : http://travis-ci.org/petemoore/build-tools/builds/41486065
15:45:52 travis-ci left the room.

We can customise this as per:
http://docs.travis-ci.com/user/notifications/#IRC-notification

<snip>

You also have the possibility to customize the message that will be sent to the channel(s) with a template:

notifications:
  irc:
    channels:
      - "chat.freenode.net#my-channel"
      - "chat.freenode.net#some-other-channel"
    template:
      - "%{repository} (%{commit}) : %{message} %{foo} "
      - "Build details: %{build_url}"

You can interpolate the following variables:

    repository_slug: your GitHub repo identifier (like svenfuchs/minimal)
    repository_name: the slug without the username
    repository: same as repository_slug [Deprecated]
    build_number: build number
    build_id: build id
    branch: branch build name
    commit: shortened commit SHA
    author: commit author name
    commit_message: commit message of build
    result: result of build
    message: travis message to the build
    duration: duration of the build
    compare_url: commit change view URL
    build_url: URL of the build detail

</snip>
Pete
Attachment #8525324 - Flags: review?(catlee)
Hey Chris (Lonnen),

How much work would it be to set up leeroybot to work around https://github.com/travis-ci/travis-ci/issues/1094 ?

Basically, we want notifications in #releng channel of all changes/test runs on all branches of the mozilla user for a given repo, but not to get notifications from any user forks.

Would it make sense to piggy-back of an existing service you are running, or set up of our infra for this?

Thanks,
Pete
Flags: needinfo?(chris.lonnen)
IRC notifications for balrog
Attachment #8525337 - Flags: review?(bhearsum)
IRC notifications for mozharness
Attachment #8525339 - Flags: review?(armenzg)
Attachment #8525337 - Flags: review?(bhearsum) → review+
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/57d006acf775f5d118737c07231def32a7768114
Bug 1076810 - enable #releng irc notifications from travis-ci for balrog
Attachment #8525339 - Flags: review?(armenzg) → review+
@pmoore -- we have our bot set up to report pretty specifically, but its easy enough to fork. Easier to work out in IRC when you have some time.
Flags: needinfo?(chris.lonnen)
To fix the three failing tests in mozharness. Essentially, nose thought they were tests, but actually they were methods used by other tests, and not directly tests themselves. If you compare to jenkins results, you will see they were not running in jenkins.

The reason they weren't running in jenkins, is because in the jenkins config, we were specifically scanning two subdirectories of mozharness, so that they would be filtered out. However, I think it is much better to be able to run nosetests out-of-the-box without having to whitelist test directories first, so i simply added the @nottest decorator to these three non-tests, to show nose that they should not be considered as tests, which also fixes the problem. It has the advantage of simplicity - running nosetests is all you have to do, rather than creating a whitelist of subdirs first.
Attachment #8527637 - Flags: review?(mgervasini)
Comment on attachment 8527637 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v1.patch

Thanks Pete!
Attachment #8527637 - Flags: review?(mgervasini) → review+
Attachment #8525324 - Flags: review?(catlee) → review?(mgervasini)
(In reply to Chris Lonnen :lonnen from comment #21)
> @pmoore -- we have our bot set up to report pretty specifically, but its
> easy enough to fork. Easier to work out in IRC when you have some time.

Thanks Chris!

I'm 9 hours behind you normally (working in Germany), so our days will not cross, but in Portland all next week - do you fancy meeting up briefly at some point next week to have a quick chat about this?

Pete
Flags: needinfo?(chris.lonnen)
Attachment #8525324 - Flags: review?(mgervasini) → review+
Comment on attachment 8527637 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v1.patch

Backed out due to mochitest bustage!

========= Started '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' warnings (results: 1, elapsed: 0 secs) (at 2014-11-24 07:15:26.000165) =========
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain-chunked --total-chunks 5 --this-chunk 1 --blob-upload-branch ash --download-symbols ondemand
 in dir /builds/slave/test/. (timeout 1800 secs) (maxTime 7200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5', '--this-chunk', '1', '--blob-upload-branch', 'ash', '--download-symbols', 'ondemand']
 environment:
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  DISPLAY=:0
  HOME=/home/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MOZ_HIDE_RESULTS_TABLE=1
  MOZ_NODE_PATH=/usr/bin/node
  MOZ_NO_REMOTE=1
  NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
  NO_FAIL_ON_TEST_ERRORS=1
  PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PROPERTIES_FILE=/builds/slave/test/buildprops.json
  PWD=/builds/slave/test
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  XDG_SESSION_COOKIE=5a152710dd62cc268aad214e000002d8-1416841705.484084-12882139
  _=/tools/buildbot/bin/python
 using PTY: False
Traceback (most recent call last):
  File "scripts/scripts/desktop_unittest.py", line 29, in <module>
    from mozharness.mozilla.testing.testbase import TestingMixin, testing_config_options
  File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 15, in <module>
    from nose.tools import nottest
ImportError: No module named nose.tools
program finished with exit code 1
elapsedTime=0.245591
========= Finished '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' warnings (results: 1, elapsed: 0 secs) (at 2014-11-24 07:15:26.780020) =========
Attachment #8527637 - Flags: checked-in+ → checked-in-
This alternative version works differently: instead of marking the /apparent/ tests as "not a test", which requires the nose libraries which are not used in mochitests, for example, instead we stop the three methods from /looking like/ a test at all, by not having "test" in the name, which means nose never considers them.

This alternative approach should be compatible with nose (so you can still run mozharness tests by issuing the nosetests command) but also does not add new dependencies for desktop_unittest.py or other test classes that inherit from TestingMixin.

This passed on my local branch:
http://travis-ci.org/petemoore/build-mozharness/builds/41984984
Attachment #8527808 - Flags: review?(armenzg)
Comment on attachment 8527808 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v2.patch

Testing in ash-mozharness with
https://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731
Comment on attachment 8527808 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v2.patch

Review of attachment 8527808 [details] [diff] [review]:
-----------------------------------------------------------------

jlund, could you please confirm/deny what I say in here?

I know what you're trying to do in here, however, I believe that breaks some assumptions for mozharness.

The prelfight and postflight methods have to match the naming of the action they cover.
http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.py#l1207

I think you need to add this (I'm not sure):
@script.PreScriptAction('run-tests')
@script.PostScriptAction('run-tests')
Attachment #8527808 - Flags: review?(armenzg) → feedback?(jlund)
@pmoore - sure thing. We'll be sharing a space so I'm sure we'll find some time. On Thursday we'll be doing a day long webdev extravaganza in the larger room and at least then we can do a BOF about setting up a hubot for your team. I'll do a little tutorial too, if you'd like.
Flags: needinfo?(chris.lonnen)
(In reply to Jordan Lund (:jlund) from comment #34)
> In production: https://hg.mozilla.org/build/mozharness/rev/9475991241fc

The backout was also merged to production, so no overall change:
https://hg.mozilla.org/build/mozharness/rev/6395cbb49840
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #32)
> Comment on attachment 8527808 [details] [diff] [review]
> bug1076810_mozharness_fix-nosetests-failures_v2.patch
> 
> Review of attachment 8527808 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> jlund, could you please confirm/deny what I say in here?
> 
> I know what you're trying to do in here, however, I believe that breaks some
> assumptions for mozharness.
> 
> The prelfight and postflight methods have to match the naming of the action
> they cover.
> http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.
> py#l1207
> 
> I think you need to add this (I'm not sure):
> @script.PreScriptAction('run-tests')
> @script.PostScriptAction('run-tests')

Thanks Armen, this was a great spot, and I hadn't noticed this requirement, but indeed you are right! Good that I chose you for the review! =)

I've found a maybe simpler way to achieve the same thing:
https://github.com/petemoore/build-mozharness/commit/1f701bdc766055e3ae92bf335192a8b523f3933b

Since this is quite a simple change, and you are sleeping at the moment, I'm send this over to Massimo for review.

Pete
Attachment #8527637 - Attachment is obsolete: true
Attachment #8527808 - Attachment is obsolete: true
Attachment #8527808 - Flags: feedback?(jlund)
Attachment #8528238 - Flags: review?(mgervasini)
Comment on attachment 8528238 [details] [diff] [review]
bug1076810_mozharness_fix-nosetests-failures_v3.patch

ship it!
Attachment #8528238 - Flags: review?(mgervasini) → review+
While I'm at it, I'll get rid of the run_travis_tests.sh file too...
Attachment #8528238 - Attachment is obsolete: true
Attachment #8528249 - Flags: review?(mgervasini)
Attachment #8528249 - Flags: review?(mgervasini) → review+
This produces breakage:
http://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731
https://tbpl.mozilla.org/php/getParsedLog.php?id=53406511&tree=Ash&full=1#error0

You can test it locally by doing this:
hg clone ash-mozharness
cd ash-mozharness
hg up -r ff58508a3731
python scripts/b2g_desktop_unittest.py --cfg b2g/desktop_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 1 --blob-upload-branch ash --download-symbols ondemand --cfg developer_config.py --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tests.zip
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #40)
> This produces breakage:
> http://hg.mozilla.org/build/ash-mozharness/rev/ff58508a3731
> https://tbpl.mozilla.org/php/getParsedLog.
> php?id=53406511&tree=Ash&full=1#error0
> 
> You can test it locally by doing this:
> hg clone ash-mozharness
> cd ash-mozharness
> hg up -r ff58508a3731
> python scripts/b2g_desktop_unittest.py --cfg
> b2g/desktop_automation_config.py --test-suite mochitest --this-chunk 1
> --total-chunks 1 --blob-upload-branch ash --download-symbols ondemand --cfg
> developer_config.py --installer-url
> http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-
> linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tar.bz2
> --test-url
> http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-
> linux32_gecko/1416935313/en-US/b2g-36.0a1.en-US.linux-i686.tests.zip

Hi Armen,

Sorry - I forgot to back that out of Ash Mozharness - will do that now. The final solution was different - this was leftover clean up from experimentation on Ash...

Pete
Ah, I see you beat me to it, many thanks!

http://hg.mozilla.org/build/ash-mozharness/rev/dbd4d9fd7627

changeset:   3683:dbd4d9fd7627
tag:         tip
user:        Armen Zambrano Gasparnian <armenzg@mozilla.com>
date:        Tue Nov 25 15:20:57 2014 -0500
summary:     Backout ff58508a3731
Hi Dustin,

Please note the first time this ran (commit e9e147cfadeaf7fb41f83601934b0d6c5ca2c267), all tests passed:
https://travis-ci.org/petemoore/build-buildbotcustom/builds/42604111

Then I added notifications (commit 96787fd26028b7d6940a0126c3dd037a0d803aaa) and all tests failed:
https://travis-ci.org/petemoore/build-buildbotcustom/builds/42605600

The change was:
$ git diff e9e147cfadeaf7fb41f83601934b0d6c5ca2c267..96787fd26028b7d6940a0126c3dd037a0d803aaa
diff --git a/.travis.yml b/.travis.yml
index 752d7de..8590a3f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -24,7 +24,7 @@ script:
 
 # currently cannot customise per user fork, see:
 # https://github.com/travis-ci/travis-ci/issues/1094
-# notifications:
-#   irc: "irc.mozilla.org#releng"
-#   on_success: always
-#   on_failure: always
+notifications:
+  irc: "irc.mozilla.org#releng"
+  on_success: always
+  on_failure: always
$ 

The notifications are unlikely to have had an effect on the test results, so I assume this is either:

1) An instability in the tests, making them unreliable
2) A problem with the travis imaging process

My guess is number 1 is much more likely.

In any case, I believe these are the correct steps to run the buildbotcustom tests, so I'm going to hope that we can solve the root cause of the above problem independently in a separate bug.

Thanks!
Pete
Attachment #8530801 - Flags: review?(dustin)
^^^^ Correction: on the second run "22/145 tests errored", not "all tests failed". ^^^^

MaxChangesHandling

testDefaultRepoBigMax ... [ERROR]
testDefaultRepoSmallMax ... [ERROR]
testMergingNoRepoBigMax ... [ERROR]
testMergingNoRepoSmallMax ... [ERROR]
testMergingNoRepoUnlimited ... [ERROR]
testNoRepoBigMax ... [ERROR]
testNoRepoSmallMax ... [ERROR]
testNoRepoUnlimited ... [ERROR]
testRelbranchSmallMax ... [ERROR]

RepoBranchHandling

testDefaultRepoBranch ... [ERROR]
testNoRepoBranch ... [ERROR]
testRelbranch ... [ERROR]

RepositoryIndexParsing

testRepositoryIndexParsing ... [ERROR]

TestPolling

testHgAllLocalesPoller ... [ERROR]
testHgLocalePoller ... [ERROR]
testHgPoller ... [ERROR]

UrlCreation

testOverrideUrl ... [ERROR]
testSimpleUrl ... [ERROR]
testTipsOnlyUrl ... [ERROR]
testTipsOnlyWithLastChangeset ... [ERROR]
testUrlWithLastChangeset ... [ERROR]
testUrlWithUnicodeLastChangeset ... [ERROR]
Comment on attachment 8530801 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v1.patch

Review of attachment 8530801 [details] [diff] [review]:
-----------------------------------------------------------------

Regarding the unstable tests .. yeah, that's certainly possible.  The incorrect version of Twisted may have something to do with it.  I don't know anything about the buildbotcustom test suite (if 'hg blame' blames me for even one line, it's long since been forgotten!), so I can't add any additional detail there.

::: .travis.yml
@@ +3,5 @@
> +  - "2.7"
> +
> +install:
> +  - pip install pycrypto
> +  - pip install twisted

I'd really recommend installing the Twisted version used in production here.  Newer Twisteds have deprecated and removed features that older Buildbots used.

I don't think the tests will be sensitive to the versions of any of the other packages here, so the existing pip installs without a version are fine.

@@ +11,5 @@
> +  - pip install pep8
> +  - pip install mock
> +  - pip install pyopenssl
> +  - pip install service-identity
> +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot

https?  And, yuck, full clones.  Is there any way to do the equivalent of --depth 50 here?

@@ +15,5 @@
> +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> +  - cd ~/buildbot/master
> +  - python setup.py install
> +  - git clone --depth 50 --branch=master git://github.com/mozilla/build-tools.git ~/tools
> +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"

weird (though harmless) quoting here..

@@ +19,5 @@
> +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> +
> +script:
> +  - mv "${TRAVIS_BUILD_DIR}" "${TRAVIS_BUILD_DIR%/*}/buildbotcustom" || true
> +  - trial --reporter=verbose $(find "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test" -name '*.py')

The find command shouldn't be necessary here -- just
  trial --reporter=verbose "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test"
should do the trick.
Attachment #8530801 - Flags: review?(dustin) → review-
Thanks Dustin! Responses below.

(In reply to Dustin J. Mitchell [:dustin] from comment #45)
> Comment on attachment 8530801 [details] [diff] [review]
> bug1076810_buildbotcustom-travis-tests_v1.patch
> 
> Review of attachment 8530801 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Regarding the unstable tests .. yeah, that's certainly possible.  The
> incorrect version of Twisted may have something to do with it.  I don't know
> anything about the buildbotcustom test suite (if 'hg blame' blames me for
> even one line, it's long since been forgotten!), so I can't add any
> additional detail there.
> 
> ::: .travis.yml
> @@ +3,5 @@
> > +  - "2.7"
> > +
> > +install:
> > +  - pip install pycrypto
> > +  - pip install twisted
> 
> I'd really recommend installing the Twisted version used in production here.
> Newer Twisteds have deprecated and removed features that older Buildbots
> used.

Thanks for the tip - I will find out which version we have, and fix it here.

> I don't think the tests will be sensitive to the versions of any of the
> other packages here, so the existing pip installs without a version are fine.
> 
> @@ +11,5 @@
> > +  - pip install pep8
> > +  - pip install mock
> > +  - pip install pyopenssl
> > +  - pip install service-identity
> > +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> 
> https?  And, yuck, full clones.  Is there any way to do the equivalent of
> --depth 50 here?

I will check if I can find a way. Or I can mirror over to github, and then use git clone --depth 50.

> 
> @@ +15,5 @@
> > +  - hg clone -u production-0.8 -b production-0.8 http://hg.mozilla.org/build/buildbot ~/buildbot
> > +  - cd ~/buildbot/master
> > +  - python setup.py install
> > +  - git clone --depth 50 --branch=master git://github.com/mozilla/build-tools.git ~/tools
> > +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> 
> weird (though harmless) quoting here..

Yes, I guess I can drop the quotes entirely. I kept the ~/ outside the quotes so that it would resolve, but since this is an assignment, technically the quotes are not needed, even if TRAVIS_BUILD_DIR contains any whitespace characters. That is just a personal habit of mine to quote variables that can contain whitespace, but in the case of assignment, word splitting does not occur on the RHS of the assignment.

To demonstrate:

# set example variable potentially containing whitespace
pmoore@Elisandra:~ $ TRAVIS_BUILD_DIR='/home/pmoore/Travis   Builds/builtbotcustom'

# without quotes
pmoore@Elisandra:~ $ echo ~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}
/Users/pmoore/tools/lib/python:/home/pmoore/Travis Builds

# with quotes around everything
pmoore@Elisandra:~ $ echo "~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
~/tools/lib/python:/home/pmoore/Travis   Builds

# with quotes around everything but not the ~/
pmoore@Elisandra:~ $ echo ~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
/Users/pmoore/tools/lib/python:/Users/pmoore/Travis   Builds

However, the 'echo' case is different to the assignment case, as the shell does not perform word splitting on the RHS of an assignment. So:

# without quotes - works
pmoore@Elisandra:~ $ PYTHONPATH=~/tools/lib/python:${TRAVIS_BUILD_DIR%/*}; echo "PYTHONPATH: '${PYTHONPATH}'"
PYTHONPATH: '/Users/pmoore/tools/lib/python:/home/pmoore/Travis   Builds'

# with quotes - also works
pmoore@Elisandra:~ $ PYTHONPATH=~"/tools/lib/python:${TRAVIS_BUILD_DIR%/*}"; echo "PYTHONPATH: '${PYTHONPATH}'"
PYTHONPATH: '~/tools/lib/python:/home/pmoore/Travis   Builds'


Therefore I'll ditch the quotes! Sorry for the long explanation!


> 
> @@ +19,5 @@
> > +  - export PYTHONPATH=~/"tools/lib/python:${TRAVIS_BUILD_DIR%/*}"
> > +
> > +script:
> > +  - mv "${TRAVIS_BUILD_DIR}" "${TRAVIS_BUILD_DIR%/*}/buildbotcustom" || true
> > +  - trial --reporter=verbose $(find "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test" -name '*.py')
> 
> The find command shouldn't be necessary here -- just
>   trial --reporter=verbose "${TRAVIS_BUILD_DIR%/*}/buildbotcustom/test"
> should do the trick.

There was no __init__.py in the buildbotcustom/test directory so passing the directory name did not work but I can add __init__.py add at the same time I'll simplify this to:
trial --reporter=verbose buildbotcustom.test

Thanks!
Pete
Added verbose mode to nosetests (-v).
Attachment #8532346 - Flags: review?(mgervasini)
Attachment #8532346 - Flags: review?(mgervasini) → review+
Attachment #8532342 - Flags: review?(rail) → review+
Comment on attachment 8532342 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v2.patch

Missing execute permissions, fixed:
https://hg.mozilla.org/build/buildbotcustom/rev/8d4acc31bdf5
This patch is in addition to previous ones (not a replacement).

It:

  * enables coverage reporting, and publishing to coveralls: https://coveralls.io/r/mozilla/build-buildbotcustom
  * enables tox to work locally, even if your checkout is not in a dir buildbotcustom (e.g. build-buildbotcustom because you clone from github)
  * adds a README.md file for presenting in github, explaining how to run tests locally
  * updates .hgignore and .gitignore to exclude test reports and coverage reports
Attachment #8532693 - Flags: review?(rail)
Attachment #8532693 - Flags: review?(rail) → review+
Improve README.md file for github viewers, and include commit message in travis notifications.
Attachment #8533176 - Flags: review?(rail)
Comment on attachment 8533176 [details] [diff] [review]
bug1076810_buildbotcustom-travis-tests_v4.patch

r+'d by rail over IRC

landed:
https://hg.mozilla.org/build/buildbotcustom/rev/2e7b5a881893
Attachment #8533176 - Flags: review?(rail) → review+
I had a gitignore before in my working repo. adding the python bits that I prefer to ignore so my 'git status' stays clean.
Attachment #8533256 - Flags: review?(pmoore)
Comment on attachment 8533256 [details] [diff] [review]
141208_bug_1076810-add_python_binaries_to_gitignore-bbotcustom.patch

Review of attachment 8533256 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks Jordan!
Attachment #8533256 - Flags: review?(pmoore) → review+
Comment on attachment 8533256 [details] [diff] [review]
141208_bug_1076810-add_python_binaries_to_gitignore-bbotcustom.patch

default: https://hg.mozilla.org/build/buildbotcustom/rev/5bcd7430ed32
Attachment #8533256 - Flags: checked-in+
Hey Massimo,

This patch provides the following changes:
1) converts mozharness testing over to tox, to align with our other repos
2) a bug fix for the issue whereby no command line options could be used when running nosetests
3) switches nosetests to used verbose and colourised logging
4) adds coverage reports, and publishes them to http://coveralls.io/
5) adds a README.md to the project, so that when viewed in github, from the front page users have an overview of the project, and useful links to find out more information, and instructions on how to test it, and run tests in travis/coveralls.
Attachment #8539236 - Flags: review?(mgervasini)
Comment on attachment 8539236 [details] [diff] [review]
bug1076810_mozharness_tox-coveralls-readme-coloured-verbose-tests_v1.patch

Thanks Pete!
Attachment #8539236 - Flags: review?(mgervasini) → review+
Comment on attachment 8539236 [details] [diff] [review]
bug1076810_mozharness_tox-coveralls-readme-coloured-verbose-tests_v1.patch

https://hg.mozilla.org/build/mozharness/rev/e110bc8e37b4
Attachment #8539236 - Flags: checked-in+
Hey Rail,

This patch:
1) Colourises the test output, making it easier to see at a glance which tests passed/failed
2) Publishes the coverage reports to coveralls (and does not cause build to fail if publish fails)
3) Adds a README.md to tools, for getting an overview when viewing in github
Attachment #8539440 - Flags: review?(rail)
Attachment #8539440 - Flags: review?(rail) → review+
We've just had a couple of failures that were actually just coveralls publishing problems. This shouldn't cause the test result to be failure, so I've moved this to "after_script" section which will not impact job result.
Attachment #8539453 - Flags: review?(rail)
Attachment #8539453 - Flags: review?(rail) → review+
Attachment #8539454 - Flags: review?(rail) → review+
Comment on attachment 8539454 [details] [diff] [review]
bug1076810_mozharness_coveralls-publish-not-to-cause-failures_v1.patch

https://hg.mozilla.org/build/mozharness/rev/701d2eda2aec
Attachment #8539454 - Flags: checked-in+
Comment on attachment 8539453 [details] [diff] [review]
bug1076810_buildbotcustom_coveralls-publish-not-to-cause-failures_v1.patch

https://hg.mozilla.org/build/buildbotcustom/rev/23df141cf12d
Attachment #8539453 - Flags: checked-in+
Added information to README.md about matching git SHAs to hg SHAs and also that we don't support github pull request workflow, with a bit more detail about process.
Attachment #8540658 - Flags: review?(mgervasini)
Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

Thanks Pete!
Attachment #8540658 - Flags: checked-in+
Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

Sorry, it was a r+ not a checked-in
Attachment #8540658 - Flags: review?(mgervasini)
Attachment #8540658 - Flags: review+
Attachment #8540658 - Flags: checked-in-
Attachment #8540658 - Flags: checked-in+
Comment on attachment 8540658 [details] [diff] [review]
bug1076810_mozharness_readme-improvements_v1.patch

https://hg.mozilla.org/build/mozharness/rev/5b95f2ee37ff
Attachment #8540658 - Flags: checked-in- → checked-in+
mozharness has been merged to production. patches are live :)
Hey Callek,

Here is my work in progress! We are hitting the pip install bug, but it sounds like you have a workaround for this, so here is my work to date. Should we land this, and then apply a separate patch for your pip fix? Or should we roll it all up into one patch?

Pete
Attachment #8544228 - Flags: review?(bugspam.Callek)
r+'d by Callek with some changes (will attach separately)
Attachment #8544228 - Attachment is obsolete: true
Attachment #8544228 - Flags: review?(bugspam.Callek)
Attachment #8544473 - Flags: review+
This is Callek's patch, which I am reviewing.
Attachment #8544474 - Flags: review?(pmoore)
Attachment #8544474 - Flags: review?(pmoore) → review+
Here is a travis irc notification from earlier today, which demonstrates the commit message missing:

travis-ci [travis-ci@moz-0eh38k.compute-1.amazonaws.com] entered the room.
travis-ci
05:38:25 mozilla/build-buildbotcustom#51 (production-0.8 - 4946124 : ffxbld): The build passed.
05:38:25 Change view : https://github.com/mozilla/build-buildbotcustom/compare/ae779e11de0c...49461241bf85
05:38:25 Build details : http://travis-ci.org/mozilla/build-buildbotcustom/builds/46032679
05:38:25 travis-ci left the room.

With this patch applied (in my personal fork) here the commit message is now included:

travis-ci [travis-ci@moz-0eh38k.compute-1.amazonaws.com] entered the room.
travis-ci
10:23:32 petemoore/build-buildbotcustom#56 (bug1076810 - 64e3406 : Pete Moore): The build passed.
10:23:32 Change view : https://github.com/petemoore/build-buildbotcustom/compare/5ae3bb94626f...64e3406662c3
10:23:32 Build details : http://travis-ci.org/petemoore/build-buildbotcustom/builds/46048372
10:23:32 Commit message : Include commit message in IRC notification.
10:23:32 travis-ci left the room.

Thanks Massimo!
Pete
Attachment #8544502 - Flags: review?(mgervasini)
Comment on attachment 8544502 [details] [diff] [review]
bug1076810_buildbotcustom_fix-irc-notifications-to-include-commit-message_v1.patch

Thanks Pete!
Attachment #8544502 - Flags: review?(mgervasini) → review+
Comment on attachment 8544502 [details] [diff] [review]
bug1076810_buildbotcustom_fix-irc-notifications-to-include-commit-message_v1.patch

https://hg.mozilla.org/build/buildbotcustom/rev/4a94dc845f16
Attachment #8544502 - Flags: checked-in+
This repo had failing builds, so I thought it would be good to add irc notifications so that it is more visible when it starts failing.

Thanks Massimo!
Pete
Attachment #8544540 - Flags: review?(mgervasini)
Comment on attachment 8544540 [details] [review]
Adding irc notifications to balrog-ui repo

https://github.com/mozilla/balrog-ui/pull/6/files looks fine, thanks Pete.
Attachment #8544540 - Flags: review?(mgervasini) → review+
Some important notes, as I am not giving this bug much attention at the moment, if anyone wants to pick up parts of it...

STILL TO DO
===========
It is worth highlighting here, that in our jenkins setup, if any of the following change:
  * buildbot-configs
  * tools
  * buildbotcustom
then we would run:
  buildbot-configs/test-masters.sh -e
Currently this is only triggered in travis for buildbot-configs changes. This means this test should also be added to the travis jobs for tools and buildbotcustom.


IRC NOTIFICATIONS
=================

clonnen also provided very useful info about how to filter irc messages using lonnenbot:
https://gist.github.com/lonnen/8699bf472c98ba34e580

I haven't done this part either yet.

As per comment 12, lonnenbot is not not needed if/when https://github.com/travis-ci/travis-ci/issues/1094 is resolved. However, if it is resolved, we should then audit all of our .travis.yml files and make sure we report consistently across our repos, e.g. that commit_message is always present in the irc notification messages, like it is here: https://github.com/mozilla/build-buildbot-configs/blob/3e8d3330620b2c7ed463372a033a6fbf650ee072/.travis.yml#L24


JORDAN'S COOL IDEAS
===================

Jordan (jlund) also noted in email:

 Then, add builder_list.py and dump_master.py runs on top of test-masters.sh. Finally, serve the results of those tests to s3 or somewhere internally within travis.

Whenever I get a buildbot r?, I run the patch with: https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_patch.sh
    e.g. $ ~/bin/dotfiles/mozilla/verify_patch.sh -e test -f $DOW/reftestchunks.patch  # apply reftestchunks.patch to bbot-cfgs repo + and get a clean/dirty copy of builder_list and dump_master diff against uni test master

If I have my own dev branch, I run the following before submitting a patch: https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_branch.sh
    e.g. $ /bin/dotfiles/mozilla/verify_branch.sh -e test -u clone_less -f clone_less  # similar to above but compare master <-> clone_less branch and use buildbot-config (-f) and buildbotcustom (-u)

we could customize scripts like these to suit our travis setup.

We shouldn't lose track of these cool ideas, so adding this to this bug, as I'm not sure how much more I'll be working on this.


TRAVIS DASHBOARD
================

Jenkins allowed us to have a simple page view of all jobs, to see which were ok/running/failing tests/erroring:
http://10.134.48.37:8080/

(access to Mozilla network required)

We miss this with travis. Something like this gives a close approximation:
http://egeloen.fr/travis-wall/#/petemoore

But it would be worth spending a little time to create a dashboard that has no personal results, and has all the RelEng repos on it, unlike the one above. Probably not a lot of work, but would help enormously, and should probably be integrated somewhere prominent in RelEng world.


SWITCHING OFF JENKINS
=====================

Probably when all of the above is done (with the possible exception of the enhancement ideas of Jordan) we can finally switch of Jenkins.
See Also: → 1118789
Adding commit message to mozharness irc notifications (to be consistent with other projects)...
Attachment #8545823 - Flags: review?(mgervasini)
Attachment #8545823 - Flags: review?(mgervasini) → review+
I spotted that tools irc messages missed the commit message:

e.g.

travis-ci
15:08:35 mozilla/build-tools#273 (master - e46c4bf : ffxbld): The build passed.
15:08:35 Change view : https://github.com/mozilla/build-tools/compare/8e1d9d06fdf3...e46c4bf47ca5
15:08:35 Build details : http://travis-ci.org/mozilla/build-tools/builds/53191909

Compare this to e.g. mozharness:
 mozilla/build-mozharness#321 (production - e4ea4fd : ffxbld): The build passed.
15:09:11 Change view : https://github.com/mozilla/build-mozharness/compare/45845c8fb610...e4ea4fd22659
15:09:11 Build details : http://travis-ci.org/mozilla/build-mozharness/builds/53193288
15:09:11 Commit message : Added FENNEC_36_0_1_RELEASE FENNEC_36_0_1_BUILD2 tag(s) for changeset production. DONTBUILD CLOSED TREE a=release


This patch adds the extra "Commit message" line.
Attachment #8573256 - Flags: review?(mgervasini)
Comment on attachment 8573256 [details] [diff] [review]
bug1076810_tools_travis-irc_v3.patch

Thanks Pete!
Attachment #8573256 - Flags: review?(mgervasini) → review+
(In reply to Pete Moore [:pete][:pmoore] from comment #97)
> Some important notes, as I am not giving this bug much attention at the
> moment, if anyone wants to pick up parts of it...
> 
> STILL TO DO
> ===========
> It is worth highlighting here, that in our jenkins setup, if any of the
> following change:
>   * buildbot-configs
>   * tools
>   * buildbotcustom
> then we would run:
>   buildbot-configs/test-masters.sh -e
> Currently this is only triggered in travis for buildbot-configs changes.
> This means this test should also be added to the travis jobs for tools and
> buildbotcustom.
> 

The outstanding bits related to buildbot become not so important in the light of moving jobs to Task Cluster. 

> 
> IRC NOTIFICATIONS
> =================
> 
> clonnen also provided very useful info about how to filter irc messages
> using lonnenbot:
> https://gist.github.com/lonnen/8699bf472c98ba34e580
> 
> I haven't done this part either yet.
> 
> As per comment 12, lonnenbot is not not needed if/when
> https://github.com/travis-ci/travis-ci/issues/1094 is resolved. However, if
> it is resolved, we should then audit all of our .travis.yml files and make
> sure we report consistently across our repos, e.g. that commit_message is
> always present in the irc notification messages, like it is here:
> https://github.com/mozilla/build-buildbot-configs/blob/
> 3e8d3330620b2c7ed463372a033a6fbf650ee072/.travis.yml#L24
> 

Since travis now have it on the roadmap to fix issue 1094, I think this isn't worth the investment now for a temporary solution, to an issue that is not so important.

> 
> JORDAN'S COOL IDEAS
> ===================
> 
> Jordan (jlund) also noted in email:
> 
>  Then, add builder_list.py and dump_master.py runs on top of
> test-masters.sh. Finally, serve the results of those tests to s3 or
> somewhere internally within travis.
> 
> Whenever I get a buildbot r?, I run the patch with:
> https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_patch.sh
>     e.g. $ ~/bin/dotfiles/mozilla/verify_patch.sh -e test -f
> $DOW/reftestchunks.patch  # apply reftestchunks.patch to bbot-cfgs repo +
> and get a clean/dirty copy of builder_list and dump_master diff against uni
> test master
> 
> If I have my own dev branch, I run the following before submitting a patch:
> https://github.com/lundjordan/dotfiles/blob/master/mozilla/verify_branch.sh
>     e.g. $ /bin/dotfiles/mozilla/verify_branch.sh -e test -u clone_less -f
> clone_less  # similar to above but compare master <-> clone_less branch and
> use buildbot-config (-f) and buildbotcustom (-u)
> 
> we could customize scripts like these to suit our travis setup.
> 
> We shouldn't lose track of these cool ideas, so adding this to this bug, as
> I'm not sure how much more I'll be working on this.
> 

This is all buildbot related, so move to Task Cluster negates the need to implement this now.

> 
> TRAVIS DASHBOARD
> ================
> 
> Jenkins allowed us to have a simple page view of all jobs, to see which were
> ok/running/failing tests/erroring:
> http://10.134.48.37:8080/
> 
> (access to Mozilla network required)
> 
> We miss this with travis. Something like this gives a close approximation:
> http://egeloen.fr/travis-wall/#/petemoore
> 
> But it would be worth spending a little time to create a dashboard that has
> no personal results, and has all the RelEng repos on it, unlike the one
> above. Probably not a lot of work, but would help enormously, and should
> probably be integrated somewhere prominent in RelEng world.
> 

Callek has kindly created:
https://wiki.mozilla.org/ReleaseEngineering/Repositories

> 
> SWITCHING OFF JENKINS
> =====================
> 
> Probably when all of the above is done (with the possible exception of the
> enhancement ideas of Jordan) we can finally switch of Jenkins.

Checking now with catlee/bhearsum if we can do this...

Based on all of the above, closing this bug now.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
QA Contact: pmoore → mshal
Commits pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/cd695e6f30c3047c3cb4f41d5141fbcb2cea0071
[balrog-ui] Bug 1076810 - RelEng CI tests should be available over the internet, add irc notifications for balrog-ui repository,r=mgerva

https://github.com/mozilla/balrog/commit/874e9f122483e0a0bb6747f5515b380280f4001e
[balrog-ui] Merge pull request #6 from petemoore/bug1076810

Bug 1076810 - RelEng CI tests should be available over the internet, add irc notifications for balrog-ui repository,r=mgerva
You need to log in before you can comment on or make changes to this bug.