1193968 - Move hgtool.py and gittool.py into external_tools/

Status: RESOLVED DUPLICATE of bug 724190

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Armen [:armenzg]]

Attachment: [mozharness] add gittool.py and hgtool.py into external_tools

In order to run Firefox UI tests on the test machines we need hgtool.py and gittool.py

Instead of deploying it, I believe it is easier and better to put them into external_tools/

I solved this with VCSTools for developer mode by downloading the files directly, however, this is now needed for normal testers.

Comment 1

[Jordan Lund (:jlund)]

Comment on attachment 8647154 [details] [diff] [review]
[mozharness] add gittool.py and hgtool.py into external_tools

Review of attachment 8647154 [details] [diff] [review]:
-----------------------------------------------------------------

bear with me as I get up to speed. This is a design change so while there is no code to review, there are a lot of implications to consider.

this looks like the copies from current puppet. I suppose the version we deploy on some of our slaves[1]. I notice that these versions are different than the source located in tools[2]. given that:

1) adding these in mozharness (and subsequently, in gecko), I fear that these will get out of date. How do you propose we keep them updated?

2) this will introduce a third place that we could be using (hg|git)tool.py. taking a quick peek here shows how we extensively use the tools copy and the slave copy (either tools copy as well from runner or the puppet copy)[3]. I could see leading to confusion for things like debugging. Thoughts?

3) at the very least, if we go with your patch, I think we should also point all mozharness references to (hg|git)tool.py to use the external_tools copy[4]. Thoughts?

4) then there is mozharness's already existing hgtool.py and gittool.py. iirc, these are wrappers for calling the tools (and now with this patch the external_tools copy). these have been out of date for a while(https://bugzil.la/724190). it was there that catlee suggested doing what you are trying to do in this patch. *and* it was you who suggested making hgtool.py a python package instead of doing this bug (https://bugzilla.mozilla.org/show_bug.cgi?id=724190#c6). Looks like there has been a lot of work in that (https://bugzil.la/1109799). Are we giving up on this effort?

I am going to f+ for now until we discuss further.

[1] http://mxr.mozilla.org/build/find?string=tool.py&tree=build&hint=%2Fpuppet%7Cmodules%7Cpackages%7Cfiles
[2] http://mxr.mozilla.org/build/find?string=%28hg%7Cgit%29tool.py&tree=build&hint=tools
[3] http://mxr.mozilla.org/build/search?string=hgtool.py http://mxr.mozilla.org/build/search?string=gittool.py
[4] http://mxr.mozilla.org/build/search?string=hgtool.py&find=mozharness&findi=&filter=%5E%5B%5E%5C0%5D*%24&hitlimit=&tree=build http://mxr.mozilla.org/build/search?string=gittool.py&find=mozharness&findi=&filter=%5E%5B%5E%5C0%5D*%24&hitlimit=&tree=build

Comment 2

[Armen [:armenzg]]

(In reply to Jordan Lund (:jlund) from comment #1)
> Comment on attachment 8647154 [details] [diff] [review]
> [mozharness] add gittool.py and hgtool.py into external_tools
> 
> Review of attachment 8647154 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> bear with me as I get up to speed. This is a design change so while there is
> no code to review, there are a lot of implications to consider.
> 
> this looks like the copies from current puppet. I suppose the version we
> deploy on some of our slaves[1]. I notice that these versions are different
> than the source located in tools[2]. given that:
> 
Correct.
The versions on the tools repo are the canonical source.
The ones on puppet are *generated* by serializing the libraries which those files import (e.g. http://mxr.mozilla.org/build/source/tools/lib/python/util/{hg|git}.py)
Generated as single files makes them not need to checkout the tools repo.

> 1) adding these in mozharness (and subsequently, in gecko), I fear that
> these will get out of date. How do you propose we keep them updated?
> 
I'm OK if they were to fall out of date.
In any case, dropping new generated version can very easily be tested by pushing to try.
It is not as bad to test as:
1) Drop a new version on puppet and push it to slaves (rollback if issues)
2) Check-in a new tools version, back out if we see an issues across trees

#1 & #2 are worse scenarioes than testing on try. It is also an idempotent deployment which can be backed out by sheriffs (if something was missed on try) w/o releng being involved or need for monitoring.

> 2) this will introduce a third place that we could be using (hg|git)tool.py.
> taking a quick peek here shows how we extensively use the tools copy and the
> slave copy (either tools copy as well from runner or the puppet copy)[3]. I
> could see leading to confusion for things like debugging. Thoughts?
> 
We should follow up with a patch making use the hgtool and gittools from external_tools.
Drop using the puppet and tools versions.

> 3) at the very least, if we go with your patch, I think we should also point
> all mozharness references to (hg|git)tool.py to use the external_tools
> copy[4]. Thoughts?
> 
Agreed.

> 4) then there is mozharness's already existing hgtool.py and gittool.py.
> iirc, these are wrappers for calling the tools (and now with this patch the
> external_tools copy). these have been out of date for a
> while(https://bugzil.la/724190). it was there that catlee suggested doing
> what you are trying to do in this patch. *and* it was you who suggested
> making hgtool.py a python package instead of doing this bug
> (https://bugzilla.mozilla.org/show_bug.cgi?id=724190#c6). Looks like there
> has been a lot of work in that (https://bugzil.la/1109799). Are we giving up
> on this effort?
> 
The already existing hgtool.py and gittool.py should use external_tools as well.
I wish someone went ahead and make that progress, however, transitioning to a python package from everywhere would be a way more involved task.
With this bug, we at least unify our current set up, add try support plus make it available for developers and testers.

> I am going to f+ for now until we discuss further.
> 
> [1]
> http://mxr.mozilla.org/build/find?string=tool.
> py&tree=build&hint=%2Fpuppet%7Cmodules%7Cpackages%7Cfiles
> [2]
> http://mxr.mozilla.org/build/find?string=%28hg%7Cgit%29tool.
> py&tree=build&hint=tools
> [3] http://mxr.mozilla.org/build/search?string=hgtool.py
> http://mxr.mozilla.org/build/search?string=gittool.py
> [4]
> http://mxr.mozilla.org/build/search?string=hgtool.
> py&find=mozharness&findi=&filter=%5E%5B%5E%5C0%5D*%24&hitlimit=&tree=build
> http://mxr.mozilla.org/build/search?string=gittool.
> py&find=mozharness&findi=&filter=%5E%5B%5E%5C0%5D*%24&hitlimit=&tree=build

Comment 4

[GitHub Autolander Bot]

Attachment: [gaia] rexboy7:1193698-remove-home-dashboard-switching > mozilla-b2g:master

Comment 5

[KM Lee [:rexboy] (inactivate after Sep. 22)]

Sorry I got the wrong bug number on my bug comment. please just ignore the patch above.