Closed
Bug 824094
Opened 11 years ago
Closed 11 years ago
Add code to run the Gaia UI tests
Categories
(Release Engineering :: General, defect, P1)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: armenzg, Assigned: armenzg)
References
Details
Attachments
(2 files, 1 obsolete file)
4.53 KB,
patch
|
mozilla
:
review+
jgriffin
:
review+
armenzg
:
checked-in+
|
Details | Diff | Splinter Review |
3.81 KB,
patch
|
mozilla
:
review+
armenzg
:
checked-in+
|
Details | Diff | Splinter Review |
/builds/panda-0082/test/build/venv/bin/python -u /builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini I have found a file that looks like gaiatest.py which is gaia_test.py and it comes from here: http://hg.mozilla.org/integration/gaia-nightly/file/default/tests/python/gaiatest/gaia_test.py 12:41:10 INFO - Running command: ['/builds/panda-0082/test/build/venv/bin/python', '-u', '/builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py', '--address', u'panda-0082:2828', '--type', 'b2g', 'tests/manifest.ini'] 12:41:10 INFO - Copy/paste: /builds/panda-0082/test/build/venv/bin/python -u /builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini 12:41:10 INFO - Traceback (most recent call last): 12:41:10 INFO - File "/builds/panda-0082/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module> 12:41:10 INFO - from gaiatest import GaiaTestCase 12:41:10 INFO - ImportError: No module named gaiatest 12:41:10 ERROR - Return code: 1
Assignee | ||
Comment 1•11 years ago
|
||
marionette.py is also no where to be found. I have tried changed line 5 to "from gaia_test" and I get this: [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini Traceback (most recent call last): File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module> from gaia_test import GaiaTestCase File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", line 9, in <module> from marionette import MarionetteTestCase ImportError: No module named marionette I tested with this gaia-tests.zip: http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121220141027/gaia-tests.zip [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0082:2828 --type b2g tests/manifest.ini Traceback (most recent call last): File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module> from gaia_test import GaiaTestCase File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", line 9, in <module> from marionette import MarionetteTestCase ImportError: No module named marionette [cltbld@foopy39 test]$ find build -name marionette.py
Comment 2•11 years ago
|
||
(In reply to Armen Zambrano G. [:armenzg] from comment #1) > marionette.py is also no where to be found. > > I have tried changed line 5 to "from gaia_test" and I get this: > [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u > /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address > panda-0082:2828 --type b2g tests/manifest.ini > Traceback (most recent call last): > File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", > line 5, in <module> > from gaia_test import GaiaTestCase > File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", > line 9, in <module> > from marionette import MarionetteTestCase > ImportError: No module named marionette > > I tested with this gaia-tests.zip: > http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ > cedar-panda/20121220141027/gaia-tests.zip > > [cltbld@foopy39 test]$ /builds/panda-0083/test/build/venv/bin/python -u > /builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py --address > panda-0082:2828 --type b2g tests/manifest.ini > Traceback (most recent call last): > File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/runtests.py", > line 5, in <module> > from gaia_test import GaiaTestCase > File "/builds/panda-0083/test/build/tests/gaiatest/gaiatest/gaia_test.py", > line 9, in <module> > from marionette import MarionetteTestCase > ImportError: No module named marionette > [cltbld@foopy39 test]$ find build -name marionette.py Marionette should now be in the gaia-tests.zip package, but the mozharness script will have to install it in a virtualenv by running Marionette's setup.py. I'm guessing that's not happening now.
Assignee | ||
Updated•11 years ago
|
Component: Marionette → Release Engineering: Automation (General)
Product: Testing → mozilla.org
QA Contact: catlee
Version: unspecified → other
Assignee | ||
Updated•11 years ago
|
Assignee: nobody → armenzg
Assignee | ||
Comment 3•11 years ago
|
||
I have tried installing gaiatest through mozharness and I don't know what I'm doing wrong. 07:50:47 INFO - Installing gaiatest into virtualenv /builds/panda-0096/test/build/venv 07:50:47 INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/gaiatest'] in /builds/panda-0096/test/build 07:50:47 INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/gaiatest 07:50:47 INFO - Ignoring indexes: http://pypi.python.org/simple/ 07:50:47 INFO - Unpacking ./tests/gaiatest 07:50:47 INFO - Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/gaiatest 07:50:47 INFO - using JS atoms from /tmp/pip-_Emi8S-build/gaiatest/atoms 07:50:47 INFO - Downloading/unpacking marionette-client (from gaiatest) 07:50:47 INFO - Could not find any downloads that satisfy the requirement marionette-client (from gaiatest) 07:50:47 INFO - No distributions at all found for marionette-client (from gaiatest) 07:50:47 INFO - Storing complete log in /home/cltbld/.pip/pip.log 07:50:47 ERROR - Return code: 1 07:50:47 FATAL - Unable to install tests/gaiatest! 07:50:47 FATAL - Exiting -1
Assignee | ||
Comment 4•11 years ago
|
||
For some reason I tried to install gaiatest instead of marionette. If I try to install marionette I get this: 08:20:54 INFO - Installing marionette into virtualenv /builds/panda-0096/test/build/venv 08:20:54 INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/marionette/client'] in /builds/panda-0096/test/build 08:20:54 INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/marionette/client 08:20:55 INFO - Ignoring indexes: http://pypi.python.org/simple/ 08:20:55 INFO - Unpacking ./tests/marionette/client 08:20:55 INFO - Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/marionette/client 08:20:55 INFO - Downloading/unpacking manifestdestiny (from marionette-client) 08:20:55 INFO - Could not find any downloads that satisfy the requirement manifestdestiny (from marionette-client) 08:20:55 INFO - No distributions at all found for manifestdestiny (from marionette-client) 08:20:55 INFO - Storing complete log in /home/cltbld/.pip/pip.log 08:20:55 ERROR - Return code: 1 08:20:55 FATAL - Unable to install tests/marionette/client! 08:20:55 FATAL - Exiting -1
Comment 5•11 years ago
|
||
manifestdestiny doesn't exist at http://puppetagain.pub.build.mozilla.org/data/python/packages/; I'll file a bug to add it.
Assignee | ||
Comment 6•11 years ago
|
||
Should I install gaiatest as well? This is where I'm at: mozbase_dir = os.path.join('tests', 'mozbase') virtualenv_modules = [ 'requests', #'mozdevice', { 'manifestparser': os.path.join(mozbase_dir, 'manifestdestiny') }, { 'mozhttpd': os.path.join(mozbase_dir, 'mozhttpd') }, { 'mozinfo': os.path.join(mozbase_dir, 'mozinfo') }, { 'mozfile': os.path.join(mozbase_dir, 'mozfile') }, { 'mozinstall': os.path.join(mozbase_dir, 'mozinstall') }, { 'mozprofile': os.path.join(mozbase_dir, 'mozprofile') }, { 'mozprocess': os.path.join(mozbase_dir, 'mozprocess') }, { 'mozrunner': os.path.join(mozbase_dir, 'mozrunner') }, { 'mozdevice': os.path.join(mozbase_dir, 'mozdevice') }, { 'marionette': os.path.join('tests', 'marionette/client') }, ] ########## 11:44:04 INFO - Installing marionette into virtualenv /builds/panda-0096/test/build/venv 11:44:04 INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/pip', 'install', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'tests/marionette/client'] in /builds/panda-0096/test/build 11:44:04 INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/pip install --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages tests/marionette/client 11:44:05 INFO - Ignoring indexes: http://pypi.python.org/simple/ 11:44:05 INFO - Unpacking ./tests/marionette/client 11:44:05 INFO - Running setup.py egg_info for package from file:///builds/panda-0096/test/build/tests/marionette/client 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): manifestdestiny in ./venv/lib/python2.7/site-packages (from marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozhttpd>=0.3 in ./venv/lib/python2.7/site-packages (from marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozprocess>=0.6 in ./venv/lib/python2.7/site-packages (from marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozrunner>=5.11 in ./venv/lib/python2.7/site-packages (from marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozdevice>=0.12 in ./venv/lib/python2.7/site-packages (from marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozinfo in ./venv/lib/python2.7/site-packages (from mozprocess>=0.6->marionette-client) 11:44:05 INFO - Requirement already satisfied (use --upgrade to upgrade): mozprofile==0.4 in ./venv/lib/python2.7/site-packages (from mozrunner>=5.11->marionette-client) 11:44:05 INFO - Installing collected packages: marionette-client 11:44:05 INFO - Running setup.py install for marionette-client 11:44:05 INFO - Installing marionette script to /builds/panda-0096/test/build/venv/bin 11:44:05 INFO - Successfully installed marionette-client 11:44:05 INFO - Cleaning up... 11:44:05 INFO - Return code: 0 11:44:05 INFO - Done creating virtualenv /builds/panda-0096/test/build/venv. 11:44:05 INFO - ##### 11:44:05 INFO - ##### Skipping request-device step. 11:44:05 INFO - ##### 11:44:05 INFO - ##### 11:44:05 INFO - ##### Running run-test step. 11:44:05 INFO - ##### 11:44:05 INFO - Getting output from command: ['/builds/panda-0096/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'] 11:44:05 INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 11:44:05 INFO - Reading from file tmpfile_stdout 11:44:05 INFO - Output received: 11:44:05 INFO - /builds/panda-0096/test/build/venv/lib/python2.7/site-packages 11:44:05 INFO - Pretend to run tests... 11:44:05 INFO - Running command: ['/builds/panda-0096/test/build/venv/bin/python', '-u', '/builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py', '--address', u'panda-0096:2828', '--type', 'b2g', 'tests/manifest.ini'] 11:44:05 INFO - Copy/paste: /builds/panda-0096/test/build/venv/bin/python -u /builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py --address panda-0096:2828 --type b2g tests/manifest.ini 11:44:05 INFO - Traceback (most recent call last): 11:44:05 INFO - File "/builds/panda-0096/test/build/tests/gaiatest/gaiatest/runtests.py", line 5, in <module> 11:44:05 INFO - from gaiatest import GaiaTestCase 11:44:05 INFO - ImportError: No module named gaiatest 11:44:05 ERROR - Return code: 1 11:44:05 INFO - ##### 11:44:05 INFO - ##### Skipping close-request step. 11:44:05 INFO - ##### 11:44:05 INFO - Copying logs to upload dir... 11:44:05 INFO - mkdir: /builds/panda-0096/test/build/upload/logs [cltbld@foopy40 test]$ ls build/tests/mozbase/ manifestdestiny mozdevice mozfile mozhttpd mozinfo mozinstall mozlog mozprocess mozprofile mozrunner README setup_development.py [cltbld@foopy40 test]$ ls build/tests gaiatest marionette mozbase
Comment 7•11 years ago
|
||
I think you just need to cd into the directory containing runtests.py before invoking it. If that doesn't work, another approach is to install gaiatest, activate the virtualenv (using bin/activate), then run the tests using 'gaiatest [args]' instead of 'python runtests.py [args]'.
Assignee | ||
Updated•11 years ago
|
Summary: Attempting to run Gaia UI tests fails to load gaiatest → Add code to run the Gaia UI tests
Assignee | ||
Comment 8•11 years ago
|
||
This worked for me. I didn't wait for all the tests to run since I wasn't sure how long I will have to wait on them. I will have them run completely by the morning.
Attachment #697468 -
Attachment is obsolete: true
Attachment #697674 -
Flags: review?(jgriffin)
Attachment #697674 -
Flags: review?(aki)
Comment 9•11 years ago
|
||
Comment on attachment 697674 [details] [diff] [review] run Gaia UI tests Nit: os.path.join(dirs['abs_gaiatest_dir'], 'tests/manifest.ini') If you're going to hardcode one '/' you can hardcode all of em :) Not a big deal. I definitely don't like the hardcoded android test zip, but I'm ok with it as a temporary stopgap. We've got some time before we either have to change the URL (due to ftp.m.o cleanup cron jobs) or get a real fix for this.
Attachment #697674 -
Flags: review?(aki) → review+
Comment 10•11 years ago
|
||
Comment on attachment 697674 [details] [diff] [review] run Gaia UI tests Review of attachment 697674 [details] [diff] [review]: ----------------------------------------------------------------- Looks good. I think the android tests.zip hack is just needed until bug 826372 lands.
Attachment #697674 -
Flags: review?(jgriffin) → review+
Assignee | ||
Comment 11•11 years ago
|
||
Comment on attachment 697674 [details] [diff] [review] run Gaia UI tests I wasn't sure what you meant in your nit but I found the proper usage of join and used it. I will have the snapshot ready in the next few hours and remove the hack.
Attachment #697674 -
Flags: checked-in+
Assignee | ||
Comment 12•11 years ago
|
||
First run: https://tbpl.mozilla.org/php/getParsedLog.php?id=18463935&tree=Cedar&full=1 07:30:29 INFO - SUMMARY 07:30:29 INFO - ------- 07:30:29 INFO - passed: 15 07:30:29 INFO - failed: 22 07:30:29 INFO - todo: 0
Assignee | ||
Comment 13•11 years ago
|
||
aki, why did the job did not go orange on tbpl? http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_panda.py#l170 170 elif code == 10: # XXX assuming this code is the right one 171 status = "test failures" 172 tbpl_status = TBPL_WARNING Do I have to return code at the end of the function?
Assignee | ||
Updated•11 years ago
|
Priority: -- → P1
Comment 14•11 years ago
|
||
(In reply to Armen Zambrano G. [:armenzg] from comment #13) > aki, why did the job did not go orange on tbpl? > > http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_panda.py#l170 > 170 elif code == 10: # XXX assuming this code is the right one > 171 status = "test failures" > 172 tbpl_status = TBPL_WARNING > > Do I have to return code at the end of the function? Ah, you didn't do anything with status or tbpl_status. http://hg.mozilla.org/build/mozharness/file/ae5e3cafedbf/scripts/desktop_unittest.py#l353 self.buildbot_status(tbpl_status, level=log_level)
Comment 15•11 years ago
|
||
(In reply to Armen Zambrano G. [:armenzg] from comment #12) > First run: > https://tbpl.mozilla.org/php/getParsedLog.php?id=18463935&tree=Cedar&full=1 > 07:30:29 INFO - SUMMARY > 07:30:29 INFO - ------- > 07:30:29 INFO - passed: 15 > 07:30:29 INFO - failed: 22 > 07:30:29 INFO - todo: 0 All of the failures relating to the Calculator app are due to the fact that the Calculator app has been removed. I need to mirror gaia-ui-tests to gaia again to fix this.
Assignee | ||
Comment 16•11 years ago
|
||
I have tested this works on staging.
Attachment #698766 -
Flags: review?(aki)
Comment 17•11 years ago
|
||
Comment on attachment 698766 [details] [diff] [review] remove hack + turn jobs orange + reduce noise on tbpl when 409 conflict appears > - self.log("Bad return status from %s: %d!" % (url, r.status_code), level=error_level) > + self.log("Bad return status from %s: %d!" % (url, r.status_code), level=warning_level) There is no such var warning_level. error_level is set at the method constructor (ERROR). Did you mean to change this? If you want it to warn, you can either self.warn(message) or change error_level to WARNING. > else: > - status = "harness failures" > - level = ERROR You could set level = INFO at the top, and keep level = ERROR here. Then if you self.buildbot_status(tbpl_status, level=level) then a failure would show up as an ERROR in the log. However, that's not a requirement, and it seems like :philor prefers not to get these messages to show up as highlights in TBPL, so this is fine. r=me if you fix the warning_level issue above.
Attachment #698766 -
Flags: review?(aki) → review+
Assignee | ||
Comment 18•11 years ago
|
||
Comment on attachment 698766 [details] [diff] [review] remove hack + turn jobs orange + reduce noise on tbpl when 409 conflict appears http://hg.mozilla.org/build/mozharness/rev/f8238b44ba61
Attachment #698766 -
Flags: checked-in+
Assignee | ||
Comment 19•11 years ago
|
||
Two things: * I would like to enable gaia ui tests on mozilla-inbound even if hidden until we go green (by disabling tests if needed) ** This way we could test the imaging process more actively * I will see if I can make a parser to get the pass/fail numbers for the TinderboxPrint
Assignee | ||
Comment 20•11 years ago
|
||
I have move the output parser work to bug 827867. This bug is completed as we're running the gaia ui tests. I will adjust the dependency of bug 826901 to block bug 802317.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•