1334209 - Activate mozharness virtualenv directly from mach commands in one-click loaners

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

When running the media mochitests (--subsuite=media), we install some extra dependencies in the mach command here:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mach_commands.py#343

But we don't install these when running them from the one-click loaner, resulting in a series of ImportErrors. We should also install this from there.

Comment 1

[Andrew Halberstadt [:ahal]]

I went to try and reproduce this, and it seemed to work fine for me out of the box. I ran option 2 at the wizard prompt, then `./mach mochitest --subsuite media`. It's mozharness that sets up the websocketprocessbridge_requirements:
https://dxr.mozilla.org/mozilla-central/rev/9c06e744b1befb3a2e2fdac7414ce18220774a1d/testing/mozharness/scripts/desktop_unittest.py#312

So choosing option 2 (mozharness setup) should get these dependencies installed into the venv, and option 2 is required to run the mach command in the first place. Paul, do you remember exactly what you took when you were getting those ImportErrors?

Comment 2

[Andrew Halberstadt [:ahal]]

*what steps you took

Comment 3

[Paul Adenot (:padenot)]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> I went to try and reproduce this, and it seemed to work fine for me out of
> the box. I ran option 2 at the wizard prompt, then `./mach mochitest
> --subsuite media`. It's mozharness that sets up the
> websocketprocessbridge_requirements:
> https://dxr.mozilla.org/mozilla-central/rev/
> 9c06e744b1befb3a2e2fdac7414ce18220774a1d/testing/mozharness/scripts/
> desktop_unittest.py#312
> 
> So choosing option 2 (mozharness setup) should get these dependencies
> installed into the venv, and option 2 is required to run the mach command in
> the first place. Paul, do you remember exactly what you took when you were
> getting those ImportErrors?

I do that multiple times per day currently, and sometimes it works, sometimes it does not work, but activating the venv works. Looks like there is a race somewhere, I don't think I proceed differently when it works and does not work.

Comment 4

[Andrew Halberstadt [:ahal]]

Ah, so this is more about a race condition causing the virtualenv not to get activated.

The current setup where the venv is sourced by the run-wizard script is fragile. I've been meaning to get the mach bootstrap script to do this in-process anyway, this seems like a good excuse.

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1334209 - Activate mozharness virtualenv in-process from one click loaner mach commands,

Previously the run-wizard script would add a command to source the virtualenv in ~/.bashrc after
mozharness finished setting things up. This is fragile, assumes people are using bash, etc. Plus
it appeared to intermittently fail for some users.

Instead, this activates the virtualenv directly from individual mach commands that need it. This
guarantees we will always be using the virtualenv if required (and won't be using it if not). The
'activate_this.py' script is invoked the same way that we do it for in-tree mach commands:
https://dxr.mozilla.org/mozilla-central/rev/9c06e744b1befb3a2e2fdac7414ce18220774a1d/python/mozbuild/mozbuild/virtualenv.py#456

Review commit: https://reviewboard.mozilla.org/r/108792/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/108792/

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8832578 [details]
Bug 1334209 - Activate mozharness virtualenv in-process from one click loaner mach commands,

https://reviewboard.mozilla.org/r/108792/#review110028

rubber stamp=me

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d70a7e18fe84
Activate mozharness virtualenv in-process from one click loaner mach commands, r=jmaher

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/d70a7e18fe84