912997 - Please schedule jit-tests on Cedar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dan Minor [:dminor]]

Once the patch for bug 858621 lands, mozharness support exists to run the jit-tests for the desktop platforms and panda android. The tests are running well locally, but scheduling the tests on Cedar will facilitate fixing any environment or platform specific problems that may exist.

I hit a few intermittent failures on pandas when testing locally, see bug 906525. Since it seems that only two tests are affected, if these show up on Cedar, it should be easy to resolve.

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: WIP patch

will test once my dev-master is finished testing the other bugs I'm working on

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: WIP patch

Comment 3

[Kim Moir [:kmoir] ET]

:dminor

Ran into this error when running tests on my dev-master today for Android jit tests on Cedar

8:43:25     INFO -  Traceback (most recent call last):
08:43:25     INFO -    File "/builds/panda-0307/test/build/tests/jit-test/jit-test/jit_test.py", line 229, in <module>
08:43:25     INFO -      main(sys.argv[1:])
08:43:25     INFO -    File "/builds/panda-0307/test/build/tests/jit-test/jit-test/jit_test.py", line 214, in main
08:43:25     INFO -      ok = jittests.run_tests_remote(job_list, prefix, options)
08:43:25     INFO -    File "/builds/panda-0307/test/build/tests/jit-test/tests/lib/jittests.py", line 628, in run_tests_remote
08:43:25     INFO -      dm.pushDir(os.path.dirname(TEST_DIR), options.remote_test_root, timeout=600)
08:43:25     INFO -  TypeError: pushDir() got an unexpected keyword argument 'timeout'

pushdir doesn't have a timeout argument

Comment 4

[Dan Minor [:dminor]]

This should have been fixed by bug 912004, so I'm guessing your're pulling in an older mozdevice. I had been pointing the virtualenv to the mozdevice in the test package in order to pick up this change.

Comment 5

[Kim Moir [:kmoir] ET]

I have patches that enable the appropriate builders on cedar.  Before I send them for review, 

terrence: do you want to schedule the jit-tests for opt or debug or both?

Comment 6

[Terrence Cole [:terrence]]

We decided on both after discussion in #ateam.

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: bug912997.patch

These patches have been testing in staging.

Regarding comments 3 and 4, we do use the mozdevice in the tests package in the virtualenv in the mozharness scripts.  From mozharness/scripts/android_panda.py  

 virtualenv_modules = [
        'mozpoolclient',
        'mozcrash',
        {'name': 'mozdevice', 'url': os.path.join('tests', os.path.join('mozbase', 'mozdevice'))}
    ]

However, in jittests.py, I think it may default to a different version of the package that's installed.  Definitely the wrong version loaded thus the error.

def run_tests_remote(tests, prefix, options):
604     # Setup device with everything needed to run our tests.
605     from mozdevice import devicemanager, devicemanagerADB, devicemanagerSUT

dminor: how did you specify the mozdevice from your tests when you were running them?

Comment 8

[Dan Minor [:dminor]]

Attachment: Dminor's local panda config file.

kmoir,

I didn't do anything special beyond the virtualenv setup you quoted above. The jittests.py harness should be running from within the virtualenv, so it should be ok.

Is it possible you are using an older tests package?

I'm attaching the panda config file I use for local testing, in case I missed something when updating the production config with my changes that might be causing this.

Comment 9

[Kim Moir [:kmoir] ET]

dminor, can you attach the script too, in addition to the config file you attached?

Comment 10

[Armen [:armenzg]]

Comment on attachment 803305 [details] [diff] [review]
bug912997.patch

Review of attachment 803305 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1341,5 @@
>                      'config_files': ["marionette/prod_config.py"],
>                  },
> +                'jittest': {
> +                    'config_files': ["unittests/mac_unittest.py"],
> +                },              

Nit: white space.

Comment 11

[Dan Minor [:dminor]]

Attachment: Dminor's shell script to run locally.

kmoir,

I'm adding the shell script I use when running locally. My android_panda.py is what is checked in to mozharness.

Sorry for the delay, but I wanted to double check this morning that things still worked for me locally. They were fine with fresh pulls from m-c and the mozharness repo.

Comment 12

[Kim Moir [:kmoir] ET]

Thanks Dan.

I think the issue is that the import in line 605 of jitests.py

  from mozdevice import devicemanager, devicemanagerADB, devicemanagerSUT

resolves to the one in /builds/sut_tools on the foopy

 13:50:11     INFO -  'PYTHONPATH': '/builds/sut_tools'

which doesn't have the timeout parameter
  def pushFile(self, localname, destname):

perhaps you need to modify the sys.path to load them from another directory from what I've read here

http://docs.python.org/2/tutorial/modules.html#the-module-search-path

Comment 13

[Dan Minor [:dminor]]

kmoir,

Does this mean that anything found under /builds/sut_tools will be taken in favour of anything installed in the virtualenv?

To save me requesting a foopy right away, do you mind changing android_panda.py to look like this (at line 232, in the _run_category_suites function):

env = self.query_env(partial_env=env, log_level=INFO)
if env.has_key('PYTHONPATH'):
    del env['PYTHONPATH']

If that doesn't work, I'll file a bug for a foopy and go from there. Thanks!

Comment 14

[Kim Moir [:kmoir] ET]

dminor, 

Yes, this is my understanding from my reading of how Python loads modules based on the search path.

I set this up on my dev-master this morning and it worked for the jittests.  Let me just run the full suite of other tests to ensure it doesn't break anything and then I can land this patch + the others to get these enabled on Cedar.

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug912997mh.patch

test results in staging look good

Comment 16

[Dan Minor [:dminor]]

Comment on attachment 804493 [details] [diff] [review]
bug912997mh.patch

Review of attachment 804493 [details] [diff] [review]:
-----------------------------------------------------------------

r+, I would just add a comment explaining that the PYTHONPATH was causing modules from outside the virtualenv to be loaded.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

This change broke talos at least on Aurora. Closing the trees.
https://tbpl.mozilla.org/php/getParsedLog.php?id=27836532&tree=Mozilla-Aurora

Comment 18

[Kim Moir [:kmoir] ET]

I think the breakage was caused by http://hg.mozilla.org/build/buildbotcustom/rev/5b6de234e2fe which is bug 829211

Comment 19

[Kim Moir [:kmoir] ET]

This was put into production on Friday.