837940 - Need mozharness script for Gaia Unit tests

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jonathan Griffin (:jgriffin)]

Once bug 837938 and bug 837936 are implemented, we need to write a mozharness script that will be used by buildbot to run the Gaia unit tests.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: WIP mozharness script for gaia-unit-tests

Here's a WIP.  It works, except it doesn't yet download the xulrunner-sdk which is needed by gaia 'make'.  I'll add that next.

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 733607 [details] [diff] [review]
WIP mozharness script for gaia-unit-tests

Review of attachment 733607 [details] [diff] [review]:
-----------------------------------------------------------------

This uses a different method of managing the gaia clone than described in bug 846384, since AFAIUI, that approach isn't viable yet (it would need at least 'git' installed on the test slaves).

I'm not sure if we want to block on bug 846384 or not; if we do, the approach currently in this script is useful for initial testing, since it works as-is.  If we definitely want to do it the way bug 846384 describes, we should figure out what we need to do to make it happen -- other than installing 'git' on the slaves, it's not clear to me what else would be required.

xre-handling code still to be added.

Comment 3

[Jonathan Griffin (:jgriffin)]

I should also add, the current script is useful for runs against m-c/m-i only; some additional work would be needed to have it run against gaia-master, so that the the correct gaia changeset is synced.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 733607 [details] [diff] [review]
WIP mozharness script for gaia-unit-tests

Hm.

I'm not a big fan of the config rewrites in _pre_config_lock() and pull().
I think we can get around them by:

a) moving the
+        if not self.config.get('gaia_dir'):
+            self.config['gaia_dir'] = self.config['base_work_dir']
   logic into query_abs_dirs(),

b) in the pull() override, we can append a gaia repo to copy.deepcopy(self.config.get('repos', [])) with commandline-overrideable url and tag.  This has the nice property of letting us easily specify different gaia repos per test or branch (since we can't easily pass in a dictionary from the commandline).  And

c) we can override create_virtualenv() to add the appropriate gaia module path to virtualenv_modules and pass that into VirtualenvMixin.create_virtualenv(modules).  Alternately call super().create_virtualenv() first and then install_module() for gaia.

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: WIP mozharness script for gaia-unit-tests

I implemented all your suggestions, and added the xre-handling code.  It all works fine (at least locally, sans the b2g desktop build problem in bug 855049).

The one thing I don't like is the way we create the gaia directory outside of the work dir, so we avoid clobbering it (which is good because cloning gaia takes a long time).  If this dir ever got into a bad state, I don't know how we'd fix it...would a buildbot clobber wipe that dir?  (It's in a sibling directory to 'build'.)

It's likely that the gittool approach may be superior if it avoids this problem, and if it works in a way that's compatible with gaia make (which generates files inside the gaia dir).  We're probably fine with the current approach for initial testing and even rollout, but may want to move to gittool as soon as it's practical (i.e., we figure out what we need to do to make that approach work and implement the appropriate pieces).

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 734082 [details] [diff] [review]
WIP mozharness script for gaia-unit-tests

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> I implemented all your suggestions, and added the xre-handling code.  It all
> works fine (at least locally, sans the b2g desktop build problem in bug
> 855049).

Thanks!

> +            print 'kwargs', kwargs

Leftover debug info?

> The one thing I don't like is the way we create the gaia directory outside
> of the work dir, so we avoid clobbering it (which is good because cloning
> gaia takes a long time).  If this dir ever got into a bad state, I don't
> know how we'd fix it...would a buildbot clobber wipe that dir?  (It's in a
> sibling directory to 'build'.)

* A clone for me took 7 minutes, which isn't outside the realm of reason to me.
Then again, I prefer taking a bit more time to make sure the env is pristine and good and trustworthy and replicatable before proceeding, whereas some developers seem to be advocating the opposite.

* I don't think a buildbot clobber will wipe the dir.  However, the MercurialVCS class should theoretically detect when it has issues and retry the clone/pull if needed, with a clobber in between.

> It's likely that the gittool approach may be superior if it avoids this
> problem, and if it works in a way that's compatible with gaia make (which
> generates files inside the gaia dir).

Yeah, hgtool and gittool are good, but I'm not sure if gittool is quite ready yet (soon, hopefully).

> We're probably fine with the current
> approach for initial testing and even rollout, but may want to move to
> gittool as soon as it's practical (i.e., we figure out what we need to do to
> make that approach work and implement the appropriate pieces).

Ok.

Comment 7

[Jonathan Griffin (:jgriffin)]

Attachment: mozharness script for gaia-unit-tests,

This implements the 'hg purge' for the gaia repo we discussed, plus has test and production config files.

Comment 8

[Jonathan Griffin (:jgriffin)]

Attachment: mozharness script for gaia-unit-tests,

Argh, forgot to delete the debugging print statement, so here's a new version.

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 736058 [details] [diff] [review]
mozharness script for gaia-unit-tests,

>+                # purge the repo if it already exists
>+                if os.access(repo_dir, os.F_OK):
>+                    cmd = [self.query_exe('hg'),
>+                           '--config',
>+                           'extensions.purge=',
>+                           'purge']
>+                    if self.run_command(cmd, cwd=repo_dir, error_list=HgErrorList):
>+                        raise VCSException("Unable to purse %s!" % dest)

sp: purge
Maybe self.fatal() instead of the exception?

Comment 10

[Jonathan Griffin (:jgriffin)]

Comment on attachment 736058 [details] [diff] [review]
mozharness script for gaia-unit-tests,

Last review comments addressed and landed as http://hg.mozilla.org/build/mozharness/rev/d372617c1842

Comment 11

[Jonathan Griffin (:jgriffin)]

Will file follow-up bugs for any additional changes needed.