836772 - Grab Gaia UI tests from Mercurial (rather than needing to mirror it into the gaia repo)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

We'll need to modify the mozharness script to clone http://hg.mozilla.org/integration/gaia-ui-tests/, install gaiatest from there (instead of gaia-tests.zip, although we'll continue to install Marionette from gaia-tests.zip), and then run the tests from the cloned repo.

Comment 1

[Armen [:armenzg]]

If someone could grab this while I work on bug 834354 it will be great.

Comment 2

[Jonathan Griffin (:jgriffin)]

I'll try to take a stab at this Tuesday.

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: Use gaia-ui-tests from cloned repo,

This seems to work ok locally, although I can only test up to the point where mozpool is invoked.This causes us to use both the tests and the gaiatest package from the cloned gaia-ui-tests repo; we still install Marionette from the --tests-url gaia-tests.zip package.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 710313 [details] [diff] [review]
Use gaia-ui-tests from cloned repo,

I think this works.

* You may want to keep virtualenv_config_options in self.config_options, though it's not required.  It mainly gives you virtualenv_path.

* I think this is a good way to keep the gaia ui tests from falling out of date, with the caveat that this is another moving target for the sheriffs to have to account for should tests start failing.  If we go down this road, we probably need to output the gaia-ui-tests revision in a TinderboxPrint.  I'm not sure how stable the repo is, but tagging or branching so that we're on a known-good set of tests might be advisable too, depending.

If, aiui, we're pretty much just hiding these if they go orange anyway, then this probably won't make things any worse, however.

Comment 5

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #4)
> Comment on attachment 710313 [details] [diff] [review]
> Use gaia-ui-tests from cloned repo,
> 
> I think this works.
> 
> * You may want to keep virtualenv_config_options in self.config_options,
> though it's not required.  It mainly gives you virtualenv_path.

testing_config_options already includes virtualenv_config_options: http://hg.mozilla.org/build/mozharness/file/6f01e94c6de1/mozharness/mozilla/testing/testbase.py#l51

> 
> * I think this is a good way to keep the gaia ui tests from falling out of
> date, with the caveat that this is another moving target for the sheriffs to
> have to account for should tests start failing.  If we go down this road, we
> probably need to output the gaia-ui-tests revision in a TinderboxPrint.  I'm
> not sure how stable the repo is, but tagging or branching so that we're on a
> known-good set of tests might be advisable too, depending.
> 
> If, aiui, we're pretty much just hiding these if they go orange anyway, then
> this probably won't make things any worse, however.

Yes, this is not a perfect solution either, but I think it will be better than having the gaia-ui-tests always be out-of-date, and requiring manual merging to gaia (which we don't have a mechanism to test).

I'll add something to output the gaia-ui-tests revision in the log.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Use gaia-ui-tests from cloned repo,

Updated with TinderboxPrint for the gaia-ui-tests revision

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 710392 [details] [diff] [review]
Use gaia-ui-tests from cloned repo,

http://hg.mozilla.org/build/mozharness/rev/34a94dd97259

I'll retrigger a run on cedar to make sure I didn't break anything.

Comment 8

[Jonathan Griffin (:jgriffin)]

And a follow-up fix: http://hg.mozilla.org/build/mozharness/rev/ab8e3afb4f5b

Comment 9

[Jonathan Griffin (:jgriffin)]

Another one: http://hg.mozilla.org/build/mozharness/rev/da68d2cbddd2

Comment 10

[Jonathan Griffin (:jgriffin)]

This works now.  All the tests are failing, but that's due to a gaia-ui-test issue that needs to be fixed.  I'll file a PR for that.