testing for mozrunner (etc)

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: k0scist, Assigned: whimboo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fixed by bug 960084])

(Reporter)

Description

6 years ago
We currently have no real way of testing for mozrunner.  Mozrunner
looks for an environment variable, BROWSER_PATH, which it can use for
the firefox location for its tests.  There is still some nuance wrt
what to do about manual testing and CI:

1. For manual testing of mozrunner, we should require a --binary
switch to point to the browser path and set this for the tests
1.a. Or if you want to run a mozrunner test individually, you can do

 BROWSER_PATH=`which firefox` python test_mozrunner.py

1.b. It would be really nice to have a mode where we don't require
Firefox, e.g. --headless or just not passing in the binary.  This will
allow quick testing of things that don't require Fx and will unblock
adding tests as the CI on k0s.org doesn't have Firefox (or X)
installed.  This will require a key in the manifests, e.g.

[DEFAULT]
require-firefox =

2. For CI, we have to figure out *what* Firefox binary to use.  There
are several solutions here:
- whatever version is on the machine
- download a new nightly each time?
- the latest build???

It would be best if mozbase tests didn't break because of Firefox.
That said, Firefox would have to be plenty broken for this to be the
case.

In any case, 2. is pretty far off.  We can do 1. today.
(Assignee)

Comment 1

6 years ago
(In reply to Jeff Hammel [:jhammel] from comment #0)
> 2. For CI, we have to figure out *what* Firefox binary to use.  There
> are several solutions here:

Why would we require Firefox specifically? What we could also do is to use a XulRunner build, which we could use to make our own mock application. That way we would have a broader coverage of Mozrunner capabilities, especially because we claim that we are compatible with all Gecko based applications.

> - whatever version is on the machine
> - download a new nightly each time?

mozdownload could be used here, which can automatically fetch the latest version for releases, candidate, and nightly builds. But we should find a way of caching the latest downloaded build. 

> - the latest build???

I think we should support all the Gecko versions we support, otherwise do we not test for backward compatibility. We already had some cases were we broke Mozmill due to code changes which were based on an API not present on the last ESR branch.

> It would be best if mozbase tests didn't break because of Firefox.
> That said, Firefox would have to be plenty broken for this to be the
> case.
> 
> In any case, 2. is pretty far off.  We can do 1. today.
(Reporter)

Comment 2

6 years ago
(In reply to Henrik Skupin (:whimboo) from comment #1)
> (In reply to Jeff Hammel [:jhammel] from comment #0)
> > 2. For CI, we have to figure out *what* Firefox binary to use.  There
> > are several solutions here:
> 
> Why would we require Firefox specifically? What we could also do is to use a
> XulRunner build, which we could use to make our own mock application. That
> way we would have a broader coverage of Mozrunner capabilities, especially
> because we claim that we are compatible with all Gecko based applications.

I think the short answer is: we wouldn't, unless we were testing FirefoxRunner specifically. Currently, FirefoxRunner accepts the `BROWSER_PATH` variable and no other runners have an equivalent which would be needed for testing, but that is easy enough to fix.  That said, we probably will want to have FirefoxRunner-specific tests and most people will have Firefox on their computer.  While for a CI-machine we can deploy however we want, I'd like to make the barrier to testing as low as possible so as to encourage development. Our tests don't require that much currently, but I've already heard multiple complaints about them (or, specifically, the mozprocess ones) not working on OOTB.

> > - whatever version is on the machine
> > - download a new nightly each time?
> 
> mozdownload could be used here, which can automatically fetch the latest
> version for releases, candidate, and nightly builds. But we should find a
> way of caching the latest downloaded build. 

For CI, mozdownload could conceivably be called from a cronjob.
 
> > - the latest build???
> 
> I think we should support all the Gecko versions we support, otherwise do we
> not test for backward compatibility. We already had some cases were we broke
> Mozmill due to code changes which were based on an API not present on the
> last ESR branch.

Again, at this point I would be happy, CI-wise, to support 1 version.  I'll agree that more == better, but don't want to block on that.

> > It would be best if mozbase tests didn't break because of Firefox.
> > That said, Firefox would have to be plenty broken for this to be the
> > case.
> > 
> > In any case, 2. is pretty far off.  We can do 1. today.
Henrik added some tests, they still aren't running in c-i (blocking on bug 966411), but that is being tracked elsewhere.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

5 years ago
Correct, that happened as part of bug 960084 for the mozrunner refactoring.
Assignee: nobody → hskupin
Depends on: 960084
Whiteboard: [fixed by bug 960084]
You need to log in before you can comment on or make changes to this bug.