Closed Bug 928366 Opened 11 years ago Closed 9 years ago

Run the Mozmill unit tests on Travis CI

Categories

(Testing Graveyard :: Mozmill, defect)

Product:
Testing Graveyard
Component:
Mozmill
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: davehunt, Assigned: whimboo)

References

Details

Attachments

(1 file, 4 obsolete files)

WIP v1
1021 bytes, patch
Details | Diff | Splinter Review 
To reduce the chances of introducing regressions in Mozmill we should create a Travis-CI job to run the unit tests. This should report failures to the #mozmill and #automation IRC channels and to the auto-tools@mozilla.com mailing list.

We would need the script to:
1. Install mozdownload using pip
2. Download the latest Firefox release using mozdownload
3. Run all the unit tests using Mutt

Travis-CI docs: http://about.travis-ci.org/docs/
mozdownload https://pypi.python.org/pypi/mozdownload
Using Mutt: https://github.com/mozilla/mozmill/blob/4b5218b01c44620bc03be93bc4c6e3184f7775cb/mutt/README.md#usage
Here's an example Travis-CI configuration file used by mozdownload: https://github.com/mozilla/mozdownload/blob/acf3e42790236cfb03bc80ec43abf1d212cce1f0/.travis.yml
:davehunt
I can take this one if you don't mind.
Assignee: nobody → jot
From what i see there's dependency conflict between mozmill which requires: mozinfo == 0.4 and latest mozdownload which needs mozinfo == 0.6.
I resolved this locally via bumping up mozinfo version for mozmill.
Also after running mutt on firefox 24.0, i'm receiving following fails (latest stable fedora, x64):
https://gist.github.com/jarekps/f260cd3d6111306d10c7
Flags: needinfo?(dave.hunt)
Hm, there shouldn't be a version conflict. I haven't seen anything when working on the mozmill-automation package for version 2.0. Are you sure that you are on the latest repository changes?
Hm, It looks like my repository isn't up-to-date with upstream. Propably my auto upstremupdate has been broken.
Okay, after updating master i don't have this requirements conflict but i'm receiving some fails during the mozmill tests:
https://travis-ci.org/jarekps/mozmill/builds/12814764

Could you have idea what i'm still missing here?
The only failure we have here is the one for the context menu:
https://travis-ci.org/jarekps/mozmill/builds/12814764#L354

This might be somewhat flaky and if it reproduces we should get it fixed.
Flags: needinfo?(dave.hunt)
:davehunt :whimboo

I'll look into this tomorrow but in a meantime could you this script: https://github.com/jarekps/mozmill/blob/bug_928366/run_tests.sh
On your environments and tell me if tests are passing ? Mhaybe i'm doing something (like missing dependencies) fundamentally wrong. On my fedora, tests are failing even on direct clone of mozilla/mozmill repo.
Which tests are failing? Do you have a list of those?
:whimboo

Looks like i've managed to eliminate some of them by running firefox in window manager under Xvfb. Finally i'm fighting with following error:
https://gist.github.com/jarekps/87e6e74ab2c59612f802
From what i see during the run, plugins tab doesn't have any plugins loaded. I've tried to find origin of this error but with no results till now. Maybe i'm overlooking something obvious :(
Please see if you have plugins installed at all on your box. It that is not the case we might have to skip this specific test by adding an additional if condition.
Wow that looks great. And it's passing! Lets add the patch here for the review process. Thanks a ton, Jarek!
Status: NEW → ASSIGNED
Okay, i've added patch.
Note here: I'm installing fluxbox as the one of lightweight X-Window managers to mimic normal user environment. It also solved some issues with context menus exceptions i had.
Attachment #825007 - Flags: review?(dave.hunt)
Comment on attachment 825007 [details] [diff] [review]
0001-Bug-928366-Add-travis-support.patch

Review of attachment 825007 [details] [diff] [review]:
-----------------------------------------------------------------

Hey Jarek, thanks so much for getting this running! I'd like to get Henrik's thoughts, but I don't think we need a run_tests.sh in this instance. If anyone wants to run the tests locally they need to have a version of Firefox installed and can simply run mutt, which is well documented.

This patch is really only for the benefit of Travis-CI which starts each run with a clean environment, so I think we just need pip install mozdownload in the install section, download the latest release in the before_script section, and then run the tests in the script section.

::: .travis.yml
@@ +1,3 @@
> +language: python
> +python:
> +      - 2.7

As we're only using one version, this can be on the line above, such as: "python: 2.7"

@@ +5,5 @@
> +before_install:
> +    - sudo apt-get update
> +    - sudo apt-get install fluxbox
> +before_script:
> +    - export DISPLAY=:99

Environment variable can be set using env: see http://about.travis-ci.org/docs/user/build-configuration/#Set-environment-variables

::: run_tests.sh
@@ +1,5 @@
> +#!/usr/bin/env bash
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this file,
> +# You can obtain one at http://mozilla.org/MPL/2.0/.
> + 

Please remove this whitespace

@@ +5,5 @@
> + 
> +VERSION_VIRTUAL_ENV="1.9.1"
> +VENV_DIR="tests/venv"
> +
> +VERSION_FIREFOX="24.0"

We shouldn't specify a Firefox version as it will become out of date. You can get the latest using: mozdownload -t release -v latest

@@ +6,5 @@
> +VERSION_VIRTUAL_ENV="1.9.1"
> +VENV_DIR="tests/venv"
> +
> +VERSION_FIREFOX="24.0"
> + 

Please remove this whitespace

@@ +19,5 @@
> +. $VENV_DIR/bin/activate
> +
> +python setup_development.py
> +
> +if [ ! -d firefox ]

We should always download the latest release version, so I don't think this is needed.
Attachment #825007 - Flags: review?(dave.hunt)
Attachment #825007 - Flags: review-
Attachment #825007 - Flags: feedback?(hskupin)
:davehunt
I've totally changed my patch and moved all parts into .travis.yml (with applying of your notes). It looks cleaner in my opinion (There's no problem to restore test script if Henrik agrees).

However, after this restructuring i've started to get javascript Exception:
Exception: "No item selected for element Elem: [object XULElement]" (/home/travis/build/jarekps/mozmill/mutt/mutt/tests/js/testElementsLib/testMenuList.js)

It looks like testXULMenuList fails on line:
menulist.select(null, alwaysActivate);

From my debug it looks like menulist has "Always activate" and there shouldn't problem with selecting this. Could you check if you can reproduce this locally or tell me which dependency could be missing?
Attachment #825007 - Attachment is obsolete: true
Attachment #825007 - Flags: feedback?(hskupin)
Attachment #826437 - Flags: review?(dave.hunt)
(In reply to Dave Hunt (:davehunt) from comment #15)
> This patch is really only for the benefit of Travis-CI which starts each run
> with a clean environment, so I think we just need pip install mozdownload in
> the install section, download the latest release in the before_script
> section, and then run the tests in the script section.

I agree with Dave here. That's all for Travis testing only.
Comment on attachment 826437 [details] [diff] [review]
0001-Bug-928366-Add-travis-support.patch

Review of attachment 826437 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Jarek Śmiejczak from comment #16)
> Created attachment 826437 [details] [diff] [review]
> 0001-Bug-928366-Add-travis-support.patch
> 
> :davehunt
> I've totally changed my patch and moved all parts into .travis.yml (with
> applying of your notes). It looks cleaner in my opinion (There's no problem
> to restore test script if Henrik agrees).

It does look much cleaner, thanks!

> However, after this restructuring i've started to get javascript Exception:
> Exception: "No item selected for element Elem: [object XULElement]"
> (/home/travis/build/jarekps/mozmill/mutt/mutt/tests/js/testElementsLib/
> testMenuList.js)
> 
> It looks like testXULMenuList fails on line:
> menulist.select(null, alwaysActivate);

Is this related to your changes? Can you replicate it locally?

Removing review request as you mention a failure you're getting with this patch applied, but giving f+ as the patch is looking good to me. Let's work out what this remaining issue is, and then we can review and land the patch.

::: .travis.yml
@@ +1,4 @@
> +language: python
> +python: 2.7
> +env: DISPLAY=:99
> +script: mutt -b firefox/firefox -m mutt/mutt/tests/manifest.ini

could you move script to below before_script so it represents the order in which the scripts will run?
Attachment #826437 - Flags: review?(dave.hunt) → feedback+
Comment on attachment 826437 [details] [diff] [review]
0001-Bug-928366-Add-travis-support.patch

Review of attachment 826437 [details] [diff] [review]:
-----------------------------------------------------------------

::: .travis.yml
@@ +10,5 @@
> +    - pip install mozdownload
> +    - python setup_development.py
> +
> +before_script:
> +    - Xvfb :99 &

I would say we use what travis proposes:
http://about.travis-ci.org/docs/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-GUI-%28e.g.-a-Web-browser%29

@@ +20,5 @@
> +#  email:
> +#      - auto-tools@lists.mozilla.org
> +#  irc:
> +#      - "irc.mozilla.org#automation"
> +#      - "irc.mozilla.org#mozmill"

This should only go to the mozmill channel.
I've updated patch after Henrik's feedback.

I mentioned my travis problems earlier. There are a few tests which failing in random manner. Some of them never ocurred on my local machine and that makes them impossible/hard to reproduce for me.
There's list of them:
https://gist.github.com/jarekps/1f76bec41a4d4aaf960f
What's worth mentioning: they always occured as a single failure and disappeared sometimes after "reset of the build". Also, i couldn't catch them after building environment similar travis-ci on my local machine.
Attachment #826437 - Attachment is obsolete: true
Attachment #828301 - Flags: feedback?(hskupin)
Comment on attachment 828301 [details] [diff] [review]
0001-Bug-928366-Add-travis-support.patch

Review of attachment 828301 [details] [diff] [review]:
-----------------------------------------------------------------

::: .travis.yml
@@ +1,3 @@
> +language: python
> +python: 2.7
> +env: DISPLAY=:99

Please move this into before_install as the travis docs mention.

@@ +4,5 @@
> +
> +before_install: sudo apt-get update
> +
> +install:
> +    - sudo apt-get install fluxbox browser-plugin-vlc

Why do we need the vlc plugin? Is it for the tests which require a plugin to be installed? If that is the case we should drop the requirement in the test. Otherwise I would propose a more popular plugin like Flash.

@@ +9,5 @@
> +    - pip install mozdownload
> +    - python setup_development.py
> +
> +before_script:
> +    - sh -e /etc/init.d/xvfb start

Please move this to before_install.

@@ +10,5 @@
> +    - python setup_development.py
> +
> +before_script:
> +    - sh -e /etc/init.d/xvfb start
> +    - fluxbox -log fluxbox.log &

Would you mind to tell me again why we need fluxbox? Is there no window manager installed by default?
Attachment #828301 - Flags: feedback?(hskupin) → feedback+
Oh and I wonder what the default screen resolution is for xvfb, and if we should set it explicitly.
> Please move this into before_install as the travis docs mention.
Do you mean to move setting environment variables? Dave already told me to use env:
> Why do we need the vlc plugin?
It's one of plugins available in default ubuntu repos. One of tests requires to install any plugin so i decided to choose open plugin instead of prioprietary flash which requires additional work of adding repositories etc. Test only enables/disables plugin.

> Would you mind to tell me again why we need fluxbox
I had some problems with bug with this contextmenu popup (mentioned earlier in gist), after installing minimal window manager it disappeared. However, i wouldn't be surprised if it was unecessary as this problem couldn't occur on my local machine.
Flags: needinfo?(hskupin)
(In reply to Jarek Śmiejczak from comment #23)
> > Please move this into before_install as the travis docs mention.
> Do you mean to move setting environment variables? Dave already told me to
> use env:

Sure but please check http://about.travis-ci.org/docs/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-GUI-%28e.g.-a-Web-browser%29

> > Why do we need the vlc plugin?
> It's one of plugins available in default ubuntu repos. One of tests requires
> to install any plugin so i decided to choose open plugin instead of
> prioprietary flash which requires additional work of adding repositories
> etc. Test only enables/disables plugin.

Why does flash require another repository? AFAIK 'flash-installer' is part of the default repositories.

> > Would you mind to tell me again why we need fluxbox
> I had some problems with bug with this contextmenu popup (mentioned earlier
> in gist), after installing minimal window manager it disappeared. However, i
> wouldn't be surprised if it was unecessary as this problem couldn't occur on
> my local machine.

I would prefer if we keep gnome installed given that this is the default window manager.
Flags: needinfo?(hskupin)
Hi, i've updated a diff a little. Now, it support gnome and flashplugin.
Attachment #828301 - Attachment is obsolete: true
Attachment #8334236 - Flags: review?(hskupin)
Comment on attachment 8334236 [details] [diff] [review]
0001-Bug-928366-Add-travis-support.patch

Review of attachment 8334236 [details] [diff] [review]:
-----------------------------------------------------------------

There is still an open question I would like to get answered first before I want to give a review on this patch.

::: .travis.yml
@@ +6,5 @@
> +    - sudo apt-get update
> +    - sh -e /etc/init.d/xvfb start
> +
> +install:
> +    - sudo apt-get install gnome-shell flashplugin-nonfree

Can you please explain why we have to install the gnome-shell package? Does the default desktop not work? Does only this fix the issues you have seen so far? If yes those might be bugs in Firefox itself for the used distribution. Which one is travis using exactly?

@@ +21,5 @@
> +#notifications:
> +    #  email:
> +      #      - auto-tools@lists.mozilla.org
> +#  irc:
> +      #      - "irc.mozilla.org#mozmill"

Why are those still commented out? Due to testing purposes?
Attachment #8334236 - Flags: review?(hskupin)
> Can you please explain why we have to install the gnome-shell package?
From the same reasons i was installing fluxbox - some of test errors disappeared. Gnome-shell was a recommended 

> Does the default desktop not work?
Propably it should, but i couldn't figure cause of errors in tests. I know it looks strange - if you see different reasons for these errors i mentioned earlier (or their random manners) i would gladly use standard xvfb window manager.

> Does only this fix the issues you have seen so far?
Not sure, because old contextmenu error started to occur again after installing nome shell.

>If yes those might be bugs in Firefox itself for the used distribution. Which one is travis using exactly?

Script downloads latest stable version via mozdownload and passes that binary to mutt. Do i need something more?
Flags: needinfo?(hskupin)
(In reply to Jarek Śmiejczak from comment #27)
> > Does the default desktop not work?
> Propably it should, but i couldn't figure cause of errors in tests. I know
> it looks strange - if you see different reasons for these errors i mentioned
> earlier (or their random manners) i would gladly use standard xvfb window
> manager.

Ok, so lets keep gnome for now, but can you please file a follow-up bug so we can investigate those issues? Also could we add an appropriate comment to the travis file?

> >If yes those might be bugs in Firefox itself for the used distribution. Which one is travis using exactly?
> 
> Script downloads latest stable version via mozdownload and passes that
> binary to mutt. Do i need something more?

Well, here I meant the Linux distribution.
Flags: needinfo?(hskupin)
>Ok, so lets keep gnome for now, but can you please file a follow-up bug so we can investigate those issues? Also could we add an appropriate comment to the travis file?

No problem :)

> Well, here I meant the Linux distribution.
At the time it surely was 24.0. Now i'm not sure if it's 25.0 already. You think that some parts of "system-installed" firefox overshadows this downloaded via mozdownload?
(In reply to Jarek Śmiejczak from comment #29)
> > Well, here I meant the Linux distribution.
> At the time it surely was 24.0. Now i'm not sure if it's 25.0 already. You
> think that some parts of "system-installed" firefox overshadows this
> downloaded via mozdownload?

New systems always can cause other behaviors. As long as I don't know which Linux distribution Travis is using, I can't really say something to it.
> New systems always can cause other behaviors. As long as I don't know which Linux distribution Travis is using, I can't really say something to it.

From docs at travis-ci.org, default build machine uses  Ubuntu 12.04 64 bit - Dave added link to the specification in one of previous messages.
If that is the case why is gnome not the default desktop manager but xvfc? That's something i don't understand.
Hm, i didn't mention xvfc but xfvb which is virtual frame buffer inside which gnome is ran.
My mistake. Sorry. Yes, I meant xfvb. So it is really necessary for gtk apps to run under xfvb or does it also work without? We never run our tests in a virtual frame buffer so far so there might be issues we don't know yet.
Jarek, do you have an update for us?
Hi Henrik, I'm really sorry for my sudden disappearance but my short honeymoon in dayjob changed into crunch-time to release product. I wanted to update my status during the week and i'm really sorry i couldn't do it by myself.
Unfortunately i'm a little too busy last days because of crunch and i just don't have a time to research it anymore.
To be precise - i'll return as soon as crunch-time will end. Now i'm a little to tired and distracted to be enough productive and i don't want to drop on quality of my contribution (as i really value your reviewing time).
I don't want to stop progress here, so please feel free to assign this task to somebody else.
Assignee: jot → nobody
Lets see what's left to do here. I will give it a try.
Assignee: nobody → hskupin
Attached patch WIP v1 — — Splinter Review 
Not sure why we are failing yet on the travis machines under xvfb. But as they write they are using:

http://docs.travis-ci.com/user/ci-environment/#CI-environment-OS
Travis CI virtual machines are based on Ubuntu 12.04 LTS Server Edition 64 bit.

I might want to setup a VM with that OS to get it tested. But first I want to fix the failures I see locally when running Mozmill under xvfb.

Also something I would like to see fixed is the output:
Xlib:  extension "RANDR" missing on display ":99.0".

We might have to request this extension to be loaded on the travis machine when starting xvfb.
Attachment #8334236 - Attachment is obsolete: true
Depends on: 973560
Depends on: 558743
Whiteboard: [mentor=davehunt][lang=py] → [mentor=davehunt][lang=py][good first bug]
Whiteboard: [mentor=davehunt][lang=py][good first bug] → [blocked by bug 558743]
Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Whiteboard: [blocked by bug 558743]
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: