991671 - Provide a simple way to run the python integration tests (aka UI Tests)

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Julien Wajsberg [:julienw]]

It's quite challenging to run the UI Tests for someone that doesn't know them well.

The goal of this bug is to take the MDN documentation [1] and turn part of it to a make goal. The make goal should be able to handle a APP= environment variable to run only the tests for one app.

[1] https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Is this a duplicate of bug 987746?

Comment 2

[Julien Wajsberg [:julienw]]

Here I was hoping for a make target in Gaia's Makefile. If I understand correctly bug 987746 would be for the B2G repository, right?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I think bug 987746 would be for mozilla-central, and should be capable of running with locally built or otherwise procured b2g/gaia builds. :ahal can correct me if I'm wrong, I'm not as familiar with mach as I should be.

Comment 4

[Julien Wajsberg [:julienw]]

Here I'm really looking for a make target because that's what we have for other tests, so the rule of least surprise would hold :)

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Julien Wajsberg [:julienw] from comment #4)
> Here I'm really looking for a make target because that's what we have for
> other tests, so the rule of least surprise would hold :)

Other gaia tests, but bug 987746 is about aligning with other mozilla tests, which may be even less surprising to contributors. I'll let :ahal respond though, as I'm already out of my depth here. :)

Comment 6

[Julien Wajsberg [:julienw]]

both works :) I bet mach could even use gaia's make ;)

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Julien Wajsberg [:julienw] from comment #6)
> both works :) I bet mach could even use gaia's make ;)

win-win :D

Comment 8

[Andrew Halberstadt [:ahal]]

(In reply to Dave Hunt (:davehunt) from comment #3)
> I think bug 987746 would be for mozilla-central, and should be capable of
> running with locally built or otherwise procured b2g/gaia builds. :ahal can
> correct me if I'm wrong, I'm not as familiar with mach as I should be.

Yes, that's right. The patch in there should work if you have a local B2G build or a local b2g desktop build (though for the latter you need to specify a gaia path).

If you want to create a make target and have the mach script call that, that would work too. Personally I'd rather see a mach-bootstrap script checked into gaia so we can just run them directly with mach and avoid two separate execution paths. I would love it in general if execution paths for running gaia tests converged with the rest of gecko :).

Comment 9

[Julien Wajsberg [:julienw]]

AFAIK mach in Gecko is a way to make running the various tools consistent. I totally see the same goal in Gaia, but that would not be my work as I don't know how to write Python :)

Comment 10

[Andrew Halberstadt [:ahal]]

Yeah, it would probably be on me :). I'd be totally fine with a make target and the mach target calling that for now.

Comment 11

[Julien Wajsberg [:julienw]]

I see that there is a name inconsistency; for example, the Messages app's directory is "sms" (and therefore we use APP=sms to run unit tests for this app) whereas it's "messages" in the UI Test's directory.

Zac, how much work would it be to match the UI Test's directory's names with the app directories?

Comment 12

[Zac C (:zac)]

I can't immediately think of a problem. We named the directories after the manifest's name which has a nice correlation with the app in the UI context.

Comment 13

[Julien Wajsberg [:julienw]]

Sorry, trying to understand properly :) do you mean you'd rather keep the tree structure as it is now, or that there should be no issue to change it? :)

Comment 14

[Zac C (:zac)]

Yes I'd rather keep it as it is now. This doesn't really seem like a P1 bug. Changing the directory is just a risk.

Comment 15

[Julien Wajsberg [:julienw]]

ok I'm glad I asked ;)

I think I'll need to have a mapping file then, I'll think about it.

Comment 16

[Zac C (:zac)]

The tests are quite flexible and can run on desktopb2g, several devices and emulator so it's hard to avoid some complexity unfortunately.

We filed to make the address default:
https://bugzilla.mozilla.org/show_bug.cgi?id=1005004
We also made a change to give default timeouts depending upon the device. These two remove two command lines required.

Also wlach is filing to run it with mach, I'm concerned there'll be 4 or more different ways to run the test and actually it will be more complicated:
https://bugzilla.mozilla.org/show_bug.cgi?id=987746

Comment 17

[Julien Wajsberg [:julienw]]

To me, mach could simply use anything we'd do here...

The mach solution will be in the B2G repository while here I wanted a purely gaia solution.

Comment 18

[Julien Wajsberg [:julienw]]

I want to have a goal in Gaia.

Comment 19

[Johan Lorenzo [:jlorenzo]]

We discussed to have this kind of feature during the past few days. I think I can provide a new make rule here.

Comment 20

[GitHub Autolander Bot]

Attachment: [gaia] JohanLorenzo:bug-991671 > mozilla-b2g:master

Comment 21

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8565400 [details] [review]
[gaia] JohanLorenzo:bug-991671 > mozilla-b2g:master

Is that the kind of Make rule you had in mind, Julien?

Comment 22

[Julien Wajsberg [:julienw]]

Comment on attachment 8565400 [details] [review]
[gaia] JohanLorenzo:bug-991671 > mozilla-b2g:master

I didn't really know about the existing script. If the script works fine I don't really need a dedicated Makefile rule, given the MDN instructions are uptodate.

I don't mind if we have one though.

Comment 23

[BMO Automation]

Firefox OS is not being worked on