880692 - Talos mozharness should download talos only once

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jason Yeo[:jyeo]]

Currently talos is downloaded twice.
1. It is cloned from http://hg.mozilla.org/build/talos.
2. It is downloaded from puppetagain and installed into virtualenv using pip.

A possible solution is to clone from the mercurial repo, and use pip to install from that local directory.

Comment 1

[Aki Sasaki (not active)]

Change the virtualenv_modules here:
http://hg.mozilla.org/build/mozharness/file/86e067433595/mozharness/mozilla/testing/talos.py#l127

If it's a string, it'll install it via whatever pypi server we're pointing at.
If it's a dict, it'll use the value as path.

kwargs['config'].setdefault(
    'virtualenv_modules', [
        {"talos": "path/to/talos"},
        "mozinstall"
    ]
)

where path/to/talos/ is a directory containing setup.py.  e.g. http://hg.mozilla.org/build/mozharness/file/86e067433595/scripts/desktop_unittest.py#l69

Comment 2

[Aki Sasaki (not active)]

(if you take this approach, we have to clone talos before we can create the virtualenv.)

Comment 3

[Jason Yeo[:jyeo]]

Attachment: Install talos from it's mercurial repo

I changed talos_url to talos_path so that it will install from the local directory instead of the url.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 759919 [details] [diff] [review]
Install talos from it's mercurial repo

a) You might want to

             virtualenv_modules = self.config.get('virtualenv_modules', [])[:]

so you don't edit your config accidentally, or this might barf when I fix bug 706036.

b) is there a reason you're editing self.config['virtualenv_modules'] rather than making it use the cloned talos every time?  Trying not to affect jetperf?

Comment 5

[Jason Yeo[:jyeo]]

(In reply to Aki Sasaki [:aki] from comment #4)
> Comment on attachment 759919 [details] [diff] [review]
> Install talos from it's mercurial repo
> 
> a) You might want to
> 
>              virtualenv_modules = self.config.get('virtualenv_modules',
> [])[:]

Okay.

> b) is there a reason you're editing self.config['virtualenv_modules'] rather
> than making it use the cloned talos every time?  Trying not to affect
> jetperf?

Not really. I'm just trying to minimize the amount of changes to the code.

Comment 6

[Jason Yeo[:jyeo]]

Attachment: Install talos from it's mercurial repo

Comment 7

[Jason Yeo[:jyeo]]

Logs show that all platforms are installing from local repository. Fixed as of http://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness/rev/de8df7a74f8b.