Closed Bug 824094 Opened 12 years ago Closed 11 years ago

Add code to run the Gaia UI tests

Categories

(Release Engineering :: General, defect, P1)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(2 files, 1 obsolete file)

run Gaia UI tests
4.53 KB, patch
mozilla
: review+
jgriffin
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
remove hack + turn jobs orange + reduce noise on tbpl when 409 conflict appears
3.81 KB, patch
mozilla
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review 
/builds/panda-0082/test/build/venv/bin/python -u /builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini

I have found a file that looks like gaiatest.py which is gaia_test.py and it comes from here:
http://hg.mozilla.org/integration/gaia-nightly/file/default/tests/python/gaiatest/gaia_test.py

12:41:10     INFO - Running command: ['/builds/panda-0082/test/build/venv/bin/python', '-u', '/builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py', '--address', u'panda-0082:2828', '--type', 'b2g', 'tests/manifest.ini']
12:41:10     INFO - Copy/paste: /builds/panda-0082/test/build/venv/bin/python -u /builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini
12:41:10     INFO -  Traceback (most recent call last):
12:41:10     INFO -    File "/builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module>
12:41:10     INFO -      from gaiatest import GaiaTestCase
12:41:10     INFO -  ImportError: No module named gaiatest
12:41:10    ERROR - Return code: 1
marionette.py is also no where to be found.

I have tried changed line 5 to "from gaia_test" and I get this:
[cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini
Traceback (most recent call last):
  File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module>
    from gaia_test import GaiaTestCase
  File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", line 9, in <module>
    from marionette import MarionetteTestCase
ImportError: No module named marionette

I tested with this gaia-tests.zip:
http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121220141027/gaia-tests.zip

[cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini
Traceback (most recent call last):
  File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module>
    from gaia_test import GaiaTestCase
  File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", line 9, in <module>
    from marionette import MarionetteTestCase
ImportError: No module named marionette
[cltbld@foopy39 test]$ find build -name marionette.py
(In reply to Armen Zambrano G. [:armenzg] from comment #1)
> marionette.py is also no where to be found.
> 
> I have tried changed line 5 to "from gaia_test" and I get this:
> [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u
> /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address
> panda-0082:2828 --type b2g tests/manifest.ini
> Traceback (most recent call last):
>   File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py",
> line 5, in <module>
>     from gaia_test import GaiaTestCase
>   File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py",
> line 9, in <module>
>     from marionette import MarionetteTestCase
> ImportError: No module named marionette
> 
> I tested with this gaia-tests.zip:
> http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/
> cedar-panda/20121220141027/gaia-tests.zip
> 
> [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u
> /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address
> panda-0082:2828 --type b2g tests/manifest.ini
> Traceback (most recent call last):
>   File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py",
> line 5, in <module>
>     from gaia_test import GaiaTestCase
>   File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py",
> line 9, in <module>
>     from marionette import MarionetteTestCase
> ImportError: No module named marionette
> [cltbld@foopy39 test]$ find build -name marionette.py

Marionette should now be in the gaia-tests.zip package, but the mozharness script will have to install it in a virtualenv by running Marionette's setup.py.  I'm guessing that's not happening now.
Component: Marionette → Release Engineering: Automation (General)
Product: Testing → mozilla.org
QA Contact: catlee
Version: unspecified → other
Assignee: nobody → armenzg
Attached patch run tests + install gaiatest (obsolete) — — Splinter Review 
I have tried installing gaiatest through mozharness and I don't know what I'm doing wrong.

07:50:47     INFO - Installing gaiatest into virtualenv /builds/panda-0096/test/build/venv
07:50:47     INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/gaiatest'] in /builds/panda-0096/test/build
07:50:47     INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/gaiatest
07:50:47     INFO -  Ignoring indexes: http://pypi.python.org/simple/
07:50:47     INFO -  Unpacking ./tests/gaiatest
07:50:47     INFO -    Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/gaiatest
07:50:47     INFO -      using JS atoms from /tmp/pip-_Emi8S-build/gaiatest/atoms
07:50:47     INFO -  Downloading/unpacking marionette-client (from gaiatest)
07:50:47     INFO -    Could not find any downloads that satisfy the requirement marionette-client (from gaiatest)
07:50:47     INFO -  No distributions at all found for marionette-client (from gaiatest)
07:50:47     INFO -  Storing complete log in /home/cltbld/.pip/pip.log
07:50:47    ERROR - Return code: 1
07:50:47    FATAL - Unable to install tests/gaiatest!
07:50:47    FATAL - Exiting -1
For some reason I tried to install gaiatest instead of marionette.
If I try to install marionette I get this:
08:20:54     INFO - Installing marionette into virtualenv /builds/panda-0096/test/build/venv
08:20:54     INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/marionette/client'] in /builds/panda-0096/test/build
08:20:54     INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/marionette/client
08:20:55     INFO -  Ignoring indexes: http://pypi.python.org/simple/
08:20:55     INFO -  Unpacking ./tests/marionette/client
08:20:55     INFO -    Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/marionette/client
08:20:55     INFO -  Downloading/unpacking manifestdestiny (from marionette-client)
08:20:55     INFO -    Could not find any downloads that satisfy the requirement manifestdestiny (from marionette-client)
08:20:55     INFO -  No distributions at all found for manifestdestiny (from marionette-client)
08:20:55     INFO -  Storing complete log in /home/cltbld/.pip/pip.log
08:20:55    ERROR - Return code: 1
08:20:55    FATAL - Unable to install tests/marionette/client!
08:20:55    FATAL - Exiting -1
manifestdestiny doesn't exist at http://puppetagain.pub.build.mozilla.org/data/python/packages/; I'll file a bug to add it.
Depends on: 826372
Should I install gaiatest as well?

This is where I'm at:
    mozbase_dir = os.path.join('tests', 'mozbase')
    virtualenv_modules = [
        'requests',
        #'mozdevice',
        { 'manifestparser': os.path.join(mozbase_dir, 'manifestdestiny') },
        { 'mozhttpd': os.path.join(mozbase_dir, 'mozhttpd') },
        { 'mozinfo': os.path.join(mozbase_dir, 'mozinfo') },
        { 'mozfile': os.path.join(mozbase_dir, 'mozfile') },
        { 'mozinstall': os.path.join(mozbase_dir, 'mozinstall') },
        { 'mozprofile': os.path.join(mozbase_dir, 'mozprofile') },
        { 'mozprocess': os.path.join(mozbase_dir, 'mozprocess') },
        { 'mozrunner': os.path.join(mozbase_dir, 'mozrunner') },
        { 'mozdevice': os.path.join(mozbase_dir, 'mozdevice') },
        { 'marionette': os.path.join('tests', 'marionette/client') },
    ]

##########
11:44:04     INFO - Installing marionette into virtualenv /builds/panda-0096/test/build/venv
11:44:04     INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/marionette/client'] in /builds/panda-0096/test/build
11:44:04     INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/marionette/client
11:44:05     INFO -  Ignoring indexes: http://pypi.python.org/simple/
11:44:05     INFO -  Unpacking ./tests/marionette/client
11:44:05     INFO -    Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/marionette/client
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): manifestdestiny in ./venv/lib/python2.7/site-packages (from marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozhttpd>=0.3 in ./venv/lib/python2.7/site-packages (from marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozprocess>=0.6 in ./venv/lib/python2.7/site-packages (from marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozrunner>=5.11 in ./venv/lib/python2.7/site-packages (from marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozdevice>=0.12 in ./venv/lib/python2.7/site-packages (from marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozinfo in ./venv/lib/python2.7/site-packages (from mozprocess>=0.6->marionette-client)
11:44:05     INFO -  Requirement already satisfied (use --upgrade to upgrade): mozprofile==0.4 in ./venv/lib/python2.7/site-packages (from mozrunner>=5.11->marionette-client)
11:44:05     INFO -  Installing collected packages: marionette-client
11:44:05     INFO -    Running setup.py install for marionette-client
11:44:05     INFO -      Installing marionette script to /builds/panda-0096/test/build/venv/bin
11:44:05     INFO -  Successfully installed marionette-client
11:44:05     INFO -  Cleaning up...
11:44:05     INFO - Return code: 0
11:44:05     INFO - Done creating virtualenv /builds/panda-0096/test/build/venv.
11:44:05     INFO - #####
11:44:05     INFO - ##### Skipping request-device step.
11:44:05     INFO - #####
11:44:05     INFO - #####
11:44:05     INFO - ##### Running run-test step.
11:44:05     INFO - #####
11:44:05     INFO - Getting output from command: ['/builds/panda-0096/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
11:44:05     INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
11:44:05     INFO - Reading from file tmpfile_stdout
11:44:05     INFO - Output received:
11:44:05     INFO -  /builds/panda-0096/test/build/venv/lib/python2.7/site-packages
11:44:05     INFO - Pretend to run tests...
11:44:05     INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/python', '-u', '/builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py', '--address', u'panda-0096:2828', '--type', 'b2g', 'tests/manifest.ini']
11:44:05     INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/python -u /builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0096:2828 --type b2g tests/manifest.ini
11:44:05     INFO -  Traceback (most recent call last):
11:44:05     INFO -    File "/builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module>
11:44:05     INFO -      from gaiatest import GaiaTestCase
11:44:05     INFO -  ImportError: No module named gaiatest
11:44:05    ERROR - Return code: 1
11:44:05     INFO - #####
11:44:05     INFO - ##### Skipping close-request step.
11:44:05     INFO - #####
11:44:05     INFO - Copying logs to upload dir...
11:44:05     INFO - mkdir: /builds/panda-0096/test/build/upload/logs
[cltbld@foopy40 test]$ ls build/tests/mozbase/
manifestdestiny  mozdevice  mozfile  mozhttpd  mozinfo  mozinstall  mozlog  mozprocess  mozprofile  mozrunner  README  setup_development.py
[cltbld@foopy40 test]$ ls build/tests
gaiatest  marionette  mozbase
I think you just need to cd into the directory containing runtests.py before invoking it.

If that doesn't work, another approach is to install gaiatest, activate the virtualenv (using bin/activate), then run the tests using 'gaiatest [args]' instead of 'python runtests.py [args]'.
Summary: Attempting to run Gaia UI tests fails to load gaiatest → Add code to run the Gaia UI tests
This worked for me.
I didn't wait for all the tests to run since I wasn't sure how long I will have to wait on them.
I will have them run completely by the morning.
Attachment #697468 - Attachment is obsolete: true
Attachment #697674 - Flags: review?(jgriffin)
Attachment #697674 - Flags: review?(aki)
Comment on attachment 697674 [details] [diff] [review]
run Gaia UI tests

Nit: os.path.join(dirs['abs_gaiatest_dir'], 'tests/manifest.ini')
If you're going to hardcode one '/' you can hardcode all of em :)
Not a big deal.

I definitely don't like the hardcoded android test zip, but I'm ok with it as a temporary stopgap.  We've got some time before we either have to change the URL (due to ftp.m.o cleanup cron jobs) or get a real fix for this.
Attachment #697674 - Flags: review?(aki) → review+
Comment on attachment 697674 [details] [diff] [review]
run Gaia UI tests

Review of attachment 697674 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.  I think the android tests.zip hack is just needed until bug 826372 lands.
Attachment #697674 - Flags: review?(jgriffin) → review+
Comment on attachment 697674 [details] [diff] [review]
run Gaia UI tests

I wasn't sure what you meant in your nit but I found the proper usage of join and used it.

I will have the snapshot ready in the next few hours and remove the hack.
Attachment #697674 - Flags: checked-in+
First run:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18463935&tree=Cedar&full=1
07:30:29     INFO -  SUMMARY
07:30:29     INFO -  -------
07:30:29     INFO -  passed: 15
07:30:29     INFO -  failed: 22
07:30:29     INFO -  todo: 0
aki, why did the job did not go orange on tbpl?

http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_panda.py#l170
   170         elif code == 10: # XXX assuming this code is the right one
   171             status = "test failures"
   172             tbpl_status = TBPL_WARNING

Do I have to return code at the end of the function?
Priority: -- → P1
(In reply to Armen Zambrano G. [:armenzg] from comment #13)
> aki, why did the job did not go orange on tbpl?
> 
> http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_panda.py#l170
>    170         elif code == 10: # XXX assuming this code is the right one
>    171             status = "test failures"
>    172             tbpl_status = TBPL_WARNING
> 
> Do I have to return code at the end of the function?

Ah, you didn't do anything with status or tbpl_status.

http://hg.mozilla.org/build/mozharness/file/ae5e3cafedbf/scripts/desktop_unittest.py#l353

self.buildbot_status(tbpl_status, level=log_level)
(In reply to Armen Zambrano G. [:armenzg] from comment #12)
> First run:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=18463935&tree=Cedar&full=1
> 07:30:29     INFO -  SUMMARY
> 07:30:29     INFO -  -------
> 07:30:29     INFO -  passed: 15
> 07:30:29     INFO -  failed: 22
> 07:30:29     INFO -  todo: 0

All of the failures relating to the Calculator app are due to the fact that the Calculator app has been removed.  I need to mirror gaia-ui-tests to gaia again to fix this.
Depends on: 826901
I have tested this works on staging.
Attachment #698766 - Flags: review?(aki)
Comment on attachment 698766 [details] [diff] [review]
remove hack + turn jobs orange + reduce noise on tbpl when 409 conflict appears

> -                    self.log("Bad return status from %s: %d!" % (url, r.status_code), level=error_level)
> +                    self.log("Bad return status from %s: %d!" % (url, r.status_code), level=warning_level)

There is no such var warning_level.  error_level is set at the method constructor (ERROR).

Did you mean to change this?  If you want it to warn, you can either self.warn(message) or change error_level to WARNING.

>          else:
> -            status = "harness failures"
> -            level = ERROR

You could set level = INFO at the top, and keep level = ERROR here.
Then if you self.buildbot_status(tbpl_status, level=level) then a failure would show up as an ERROR in the log.

However, that's not a requirement, and it seems like :philor prefers not to get these messages to show up as highlights in TBPL, so this is fine.


r=me if you fix the warning_level issue above.
Attachment #698766 - Flags: review?(aki) → review+
Comment on attachment 698766 [details] [diff] [review]
remove hack + turn jobs orange + reduce noise on tbpl when 409 conflict appears

http://hg.mozilla.org/build/mozharness/rev/f8238b44ba61
Attachment #698766 - Flags: checked-in+
Two things:
* I would like to enable gaia ui tests on mozilla-inbound even if hidden until we go green (by disabling tests if needed)
** This way we could test the imaging process more actively

* I will see if I can make a parser to get the pass/fail numbers for the TinderboxPrint
I have move the output parser work to bug 827867.
This bug is completed as we're running the gaia ui tests.
I will adjust the dependency of bug 826901 to block bug 802317.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
No longer depends on: 826901
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: