796200 - Make a separate package for the Gaia test class

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

Now that we have a separate test class for Gaia tests, we should package this and upload it to pypi.  Then, running a Gaia test will be as easy as:

1. easy_install gaiatest
2. gaiatest [args] /path/to/test_something.py

Comment 1

[Jonathan Griffin (:jgriffin)]

I have a patch for this but am not putting it up for review until we get the process sorted out, since it's unclear who should be required to approve this and where (here or github).

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Pointer to pull request

Comment 3

[Jonathan Griffin (:jgriffin)]

@mdas, when you r+ this, can you indicate that in the PR as well?

Comment 4

[Jonathan Griffin (:jgriffin)]

Request blocking-basecamp status; this change is needed to make it easier to run Python Marionette Gaia tests.

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 670024 [details]
Pointer to pull request

Looks good to me, I just had a few suggestions that I added as comments on the PR

Comment 6

[Dietrich Ayala (:dietrich)]

didn't forget about this - i'm not sure blocking matters at all for infrastructure changes like this. i've asked the module owners to add clear NPOTB policy to https://wiki.mozilla.org/Tree_Rules/B2G.

my opinion are that these shouldn't need blocking flags at all, and my advice would be to land and ping vivien/cjones to make them aware of the landing.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Tests are currently NPOTB because we're failing to run them on main buildbot infra.  We don't *want* tests to be NPOTB.

I am fully in favor of landing tests without blocking-basecamp+, but the tests need to be reviewed.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #7)
> Tests are currently NPOTB because we're failing to run them on main buildbot
> infra.  We don't *want* tests to be NPOTB.
> 
> I am fully in favor of landing tests without blocking-basecamp+, but the
> tests need to be reviewed.

We (Web QA + core Gaia-QA team) aim to be running these soon, and will naturally have to go through test-stability/coverage, etc. questions.  As I've said a couple times before, we don't anticipate them blocking any development for a while -- at least, again, not until we've proven the technology and coverage is where we want and need it to be.

Chris, Vivien, by whom would the tests (again, these are Python, because that's my team's expertise) need to be reviewed?  Is module-owner for each app good enough?  The aim here is to get QA coverage and alleviate the burden of running the smoketests, not introduce yet-another-blocker for development, which I already understand, from Vivien, is that devs aren't (not should they be) interested in learning even enough Python to review these tests.

Comment 9

[Jonathan Griffin (:jgriffin)]

This particular bug is more about landing test harness changes, rather than tests, anyway.

It would be nice to able to land test harness changes with peer review, but without further approvals.  Having test harness fixes take a week to get approval (https://github.com/mozilla-b2g/gaia/pull/5750) really impacts our ability to deliver code efficiently.

Comment 10

[Jonathan Griffin (:jgriffin)]

Landed on Gaia:  https://github.com/mozilla-b2g/gaia/commit/1950a262af34417ebb3c2e518e2c31e34b0c3720

Comment 11

[Andrew Overholt [:overholt]]

Clearing unnecessary blocker flags.