1229713 - mach talos-test downloads all its things into the tree and clutters up hg st with lots of unknown files

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[:Gijs (he/him)]

And when I say lots, I mean lots:

$ hg st -un | grep mozharness | wc -l
952

almost 1000 files? That is not OK - this stuff should go in a . directory or at least be added to .hgignore.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

right now ./mach talos-test run mozharness style to the point of much wasted work/files.  We could make this smarter by knowing we are in tree and using the resources available to us better.  Talos still needs a virtualenv, but that can be put in the objdir instead of the srcdir.

thanks for filing this!

Comment 2

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

There's always ~/.mozbuild/

Comment 3

[Julien Pagès (:parkouss)]

I believe most of these files are there because we use mozharness to run talos locally. And mozharness takes care of creating the virtualenv, but I noticed it does it inside the source dir.

So I can see three options so far:

- change mozharness to not do that (not sure how)
- add the virtualenv dir created by mozharness in .hgignore
- change talos to not use mozharness when runing locally from mach (I know :jmaher does not really like that idea, to be closer to what is run on try)

Also there will be a few files created by talos runs - those would need to be handled differently (bug 735506).

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I prefer to keep the mozharness method of running the tests, but would be open to doing this more streamlined.

most likely if we can put the virtualenv elsewhere, then we don't need to worry about much else.

Comment 5

[Gregory Szorc [:gps]]

Please don't clutter the source directory - ever. If I had my way, I'd forbid objdirs from existing in the source directory. ~/.mozbuild, objdir, or temporary directories are where things should go. We don't yet have a good logical place for state tied to a specific source directory [and shared across many objdirs]. We should invent said location (probably under ~/.mozbuild somewhere by hashing the source path or something). But that's likely for another bug.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:parkouss, we should tackle this bug soon. When I get the tart crash resolved I can do this- likewise you could if you have some cycles.

Comment 7

[Julien Pagès (:parkouss)]

Sure, I take it - just looked in the code and saw that it is easy to put everything we create (virtual env and a config file) in $objdir/testing/.

BTW, forgot my comment 3 - this is not a mozharness issue at all. ;

Comment 8

[Julien Pagès (:parkouss)]

Attachment: MozReview Request: Bug 1229713 - mach talos-test downloads all its things into the source tree. r=jmaher

Review commit: https://reviewboard.mozilla.org/r/28069/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/28069/

Comment 9

[Julien Pagès (:parkouss)]

:jmaher, feel free to redirect the review flag if needed (maybe :gps or :armenzg ?).

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8698682 [details]
MozReview Request: Bug 1229713 - mach talos-test downloads all its things into the source tree. r=jmaher

https://reviewboard.mozilla.org/r/28069/#review25171

::: testing/talos/mach_commands.py:41
(Diff revision 1)
> -        self.config_file_path = os.path.join(self.config_dir,
> +                                             'talos-in_tree_conf.json')

I was under the impression this is 'in_tree_config.json' without the 'talos-' at the front.

::: testing/talos/mach_commands.py:45
(Diff revision 1)
> -        self.virtualenv_path = os.path.join(self.mozharness_dir, 'venv')
> +        self.virtualenv_path = os.path.join(self._topobjdir, 'testing',

is self._topobjdir defined and realistic in the buildbot automation?

a few things, lets sort them out first.

Comment 11

[Julien Pagès (:parkouss)]

I forgot to say that it works for me, I tested. mach_commands is not used in automation - this is only for local runs.

Also, 'talos-in_tree_conf.json' is an arbitrary name for the config. I just wanted to prefix 'talos-' to this file since it is in the objdir/testing directory.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8698682 [details]
MozReview Request: Bug 1229713 - mach talos-test downloads all its things into the source tree. r=jmaher

https://reviewboard.mozilla.org/r/28069/#review25175

thanks parkouss!  then lets ship it

Comment 13

[Julien Pagès (:parkouss)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/67730d70a0f296dab566b0ac7f68de3aeb409de1
Bug 1229713 - mach talos-test downloads all its things into the source tree. r=jmaher

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/67730d70a0f2