946649 - Schedule gaia-ui-tests on cedar against emulator builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

These are not currently expected to pass, but we should get them enabled to see the state of them. Bug 946644 covers the work needed to get the tests into shape before we can consider turning them on for other branches.

Note that for emulator we will need the --sdcard command line argument with a sensible value. I've been using 1G locally, but I suspect 512M would also be enough.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Jonathan, could you help with this? I'm happy to write the patch to enable this with some guidance.

Comment 2

[Jonathan Griffin (:jgriffin)]

I'll handle the buildbot patch for this.  I know dburns was trying to get these to run locally and was having problems; David, did you ever figure out what was wrong?  According to davehunt, this was working a few weeks ago, but it's possible something regressed.

Comment 3

[David Burns :automatedtester]

When I run the tests I get 

(gaiatest)☁  gaia-ui-tests [34688bb] ⚡ gaiatest --emulator=arm --homedir=~/emulator --restart --sdcard=1GB gaiatest/gaiatest/tests/functional/dialer/ --testvars gaiatest/testvars.json
starting httpd
running webserver on http://172.16.17.240:35501/
waiting for system-message-listener-ready...
done
waiting for homescreen...
...done

SUMMARY
-------
passed: 0
failed: 0
todo: 0
Traceback (most recent call last):
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/runner/base.py", line 731, in run_tests
    self.marionette.check_for_crash()
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/marionette.py", line 658, in check_for_crash
    if self.emulator.check_for_minidumps():
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/emulator.py", line 168, in check_for_minidumps
    return self.b2g.check_for_crashes()
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/b2ginstance.py", line 57, in check_for_crashes
    remote_dump_dirs = [posixpath.join(p, 'minidumps') for p in self.remote_profiles]
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/b2ginstance.py", line 34, in remote_profiles
    self.check_remote_profiles()
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/b2ginstance.py", line 38, in check_remote_profiles
    if not self.dm.fileExists(remote_profiles_ini):
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/marionette_client-0.7.2-py2.7.egg/marionette/b2ginstance.py", line 28, in dm
    self._dm = DeviceManagerADB(adbPath=self.adb_path)
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/mozdevice-0.31-py2.7.egg/mozdevice/devicemanagerADB.py", line 66, in __init__
    self._verifyDevice()
  File "/home/dburns/.virtualenvs/gaiatest/local/lib/python2.7/site-packages/mozdevice-0.31-py2.7.egg/mozdevice/devicemanagerADB.py", line 653, in _verifyDevice
    raise DMError("unable to connect to device")
DMError: unable to connect to device


The emulator starts up and then this is what happens.

I have created a new virtualenv and run python setup.py develop from 

(gaiatest)☁  gaia-ui-tests [34688bb] ⚡ pwd
/home/dburns/emulator/gaia/tests/python/gaia-ui-tests

Comment 4

[Jonathan Griffin (:jgriffin)]

Ah, looks like we've regressed on something.  Thanks, David, I'll take  a look at this next week.

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaia-ui-tests on emulators on cedar

This will do the actual scheduling, but first we'll need a new mozharness config file, and potentially some changes to the mozharness script as well.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Add emulator support to gaiatest mozharness script,

This will go together with a buildbot patch that causes us to run the script with --cfg gaia_ui_test_prod_config.py --cfg gaia_ui_test_emu_config.py.

Comment 7

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaiatest on emulators on cedar

I'll test the mozharness patch on ash first, then land it, then land the buildbot patch, if all goes well.

Comment 8

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaia-ui-tests on emulators on cedar,

Grumble, the last patch was empty...

Comment 9

[Andrew Halberstadt [:ahal]]

Comment on attachment 8360104 [details] [diff] [review]
Add emulator support to gaiatest mozharness script,

Review of attachment 8360104 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, lgtm.

::: configs/marionette/gaia_ui_test_emu_config.py
@@ +1,2 @@
> +# This is a config file for gaiatest on emulators; it's meant to be used
> +# with the primary gaia-ui-test config file.

I think we could do "from gaia_ui_test import config" and then config.update to avoid needing to pass both config files in on the command line.

Comment 10

[Jonathan Griffin (:jgriffin)]

> I think we could do "from gaia_ui_test import config" and then config.update to avoid needing to pass 
> both config files in on the command line.

We can't actually, since the directory that file is in is not in sys.path...there are probably ways around that limitation (like hacking sys.path with the directory of any cfg files passed to mozharness), but I'm not sure it's worth doing.

I'm going to run this on cypress to make sure I didn't bust anything.

Comment 11

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/build/mozharness/rev/332cc105d866

Comment 12

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #11)
> https://hg.mozilla.org/build/mozharness/rev/332cc105d866

This looks good on cedar/cypress.

Comment 13

[Chris AtLee [:catlee]]

(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> (In reply to Jonathan Griffin (:jgriffin) from comment #11)
> > https://hg.mozilla.org/build/mozharness/rev/332cc105d866
> 
> This looks good on cedar/cypress.

merged to production

Comment 14

[Jonathan Griffin (:jgriffin)]

(In reply to Chris AtLee [:catlee] from comment #13)
> (In reply to Jonathan Griffin (:jgriffin) from comment #12)
> > (In reply to Jonathan Griffin (:jgriffin) from comment #11)
> > > https://hg.mozilla.org/build/mozharness/rev/332cc105d866
> > 
> > This looks good on cedar/cypress.
> 
> merged to production

backed out; it caused Gu to fail with a permissions error:

https://tbpl.mozilla.org/php/getParsedLog.php?id=33115025&tree=B2g-Inbound&full=1#error0

Comment 15

[Jonathan Griffin (:jgriffin)]

This patch caused us to pass two --binary parameters to Gu, one of them incorrect!  I'll test a fix out on ash later this afternoon.

Comment 16

[Jonathan Griffin (:jgriffin)]

Attachment: Add emulator support to gaiatest mozharness script,

I made a dumb mistake; this fixes it.  I'm running it on ash to confirm.

Comment 17

[Jonathan Griffin (:jgriffin)]

Ash looks fine; landed at https://hg.mozilla.org/build/mozharness/rev/d416937ec90a

Comment 18

[John Hopkins (:jhopkins)]

mozharness merged to production

Comment 19

[Jonathan Griffin (:jgriffin)]

buildbot-config patch: https://hg.mozilla.org/build/buildbot-configs/rev/e5abee747f33

Comment 20

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #19)
> buildbot-config patch:
> https://hg.mozilla.org/build/buildbot-configs/rev/e5abee747f33

backed out for a typo; re-landed as https://hg.mozilla.org/build/buildbot-configs/rev/7c742fd377c3

Comment 21

[John Hopkins (:jhopkins)]

in production.

Comment 22

[Jonathan Griffin (:jgriffin)]

These are scheduled but failing; looks like we need to define tooltool_servers in the config file now.

Comment 23

[Jonathan Griffin (:jgriffin)]

Attachment: Add tooltool_servers to gaia-ui emu config,

Comment 24

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8366139 [details] [diff] [review]
Add tooltool_servers to gaia-ui emu config,

https://hg.mozilla.org/build/mozharness/rev/f27754240526

Comment 25

[Jonathan Griffin (:jgriffin)]

That didn't quite do it.  It looks like the issue is that emulator scripts default to the old method of updating gecko/gaia after launching the emulator.  We don't run tests this way any longer; the other scripts are invoked with a --no-update argument by buildbot to avoid this behavior.

Ahal, is there any reason to keep this logic around any longer?  I'm tempted to get rid of it altogether, but in the meantime, I'll add "no_update" to the mozharness config here.

Comment 26

[Jonathan Griffin (:jgriffin)]

Attachment: Invoke gaiatest on emulators with no_update,

We don't need to define tooltool_servers after all, since that's only needed for old-style emulator installs, which we don't do any longer.

Comment 27

[Andrew Halberstadt [:ahal]]

(In reply to Jonathan Griffin (:jgriffin) from comment #25)
> That didn't quite do it.  It looks like the issue is that emulator scripts
> default to the old method of updating gecko/gaia after launching the
> emulator.  We don't run tests this way any longer; the other scripts are
> invoked with a --no-update argument by buildbot to avoid this behavior.
> 
> Ahal, is there any reason to keep this logic around any longer?  I'm tempted
> to get rid of it altogether, but in the meantime, I'll add "no_update" to
> the mozharness config here.

They do? That's news to me..

Comment 28

[Jonathan Griffin (:jgriffin)]

(In reply to Andrew Halberstadt [:ahal] from comment #27)
> (In reply to Jonathan Griffin (:jgriffin) from comment #25)
> > That didn't quite do it.  It looks like the issue is that emulator scripts
> > default to the old method of updating gecko/gaia after launching the
> > emulator.  We don't run tests this way any longer; the other scripts are
> > invoked with a --no-update argument by buildbot to avoid this behavior.
> > 
> > Ahal, is there any reason to keep this logic around any longer?  I'm tempted
> > to get rid of it altogether, but in the meantime, I'll add "no_update" to
> > the mozharness config here.
> 
> They do? That's news to me..

Indeed, see e.g., http://hg.mozilla.org/build/buildbot-configs/file/02874ef2b882/mozilla-tests/b2g_config.py#l915

Comment 29

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8366246 [details] [diff] [review]
Invoke gaiatest on emulators with no_update,

https://hg.mozilla.org/build/mozharness/rev/d04ed03a6678

Comment 30

[Jordan Lund (:jlund)]

mozharness patch: 'Add tooltool_servers to gaia-ui emu config', was merged into production as of ~3pm PT! :)

Comment 31

[Jonathan Griffin (:jgriffin)]

Attachment: no_update -> update_files,

:sigh:, this should have been update_files = False, not no_update = True

Comment 32

[Andrew Halberstadt [:ahal]]

Comment on attachment 8367666 [details] [diff] [review]
no_update -> update_files,

Review of attachment 8367666 [details] [diff] [review]:
-----------------------------------------------------------------

I'll take your word for it :p

Comment 33

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8367666 [details] [diff] [review]
no_update -> update_files,

https://hg.mozilla.org/build/mozharness/rev/19a6a43e0231

Comment 34

[Aki Sasaki (not active)]

In production.

Comment 35

[Jonathan Griffin (:jgriffin)]

The tests are running, but failing...but that's for another bug!

Comment 36

[Julien Wajsberg [:julienw]]

Jonathan, is there another bug already ?

Zac is saying we should probably use --timeout 60000 instead of 10000 for emulator builds.

Comment 37

[Dave Hunt [:davehunt] [he/him] ⌚BST]

See the blocked bugs: bug 916368 and bug 946644