1030679 - Make Gaia UI tests runs Mulet

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[:gerard-majax]

Let's make it work.

Comment 1

[Zac C (:zac)]

Can we get a nightly binary from pvt/releng yet?

Comment 2

[Alexandre Poirot [:ochameau]]

It's not ready yet, you have to build it from mozilla-central with 
  ac_add_options --enable-application=b2g/dev
in your mozconfig.

Having tests running and being green is a condition to have nightly builds.

Comment 3

[:gerard-majax]

(In reply to Zac C (:zac) from comment #1)
> Can we get a nightly binary from pvt/releng yet?

I think it's in the process of getting built on tbpl. Meanwhile, I'm pulling new gecko master and rebuilding, I can share the result :)

Comment 4

[Alexandre Poirot [:ochameau]]

Yes it is already being built in Fig branch over here:
  https://tbpl.mozilla.org/?tree=Fig
And possibly try builds also.

Comment 5

[Zac C (:zac)]

OK thanks Alexandre. With access to pre-built binaries we can run tests locally if/when you need us to.

Comment 6

[:gerard-majax]

One will need also the fix from bug 1016030: https://github.com/ochameau/mozilla-central/commit/1bdb789f58847347b627193a5fbec2dc6766b852

Comment 7

[:gerard-majax]

Until we fix mozrunner/gaiarunner,

Produce a profile:
> $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests

Run mulet:
> $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html

Please make sure it has been built with marionette, of course.

Then start the tests:
> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log

Comment 8

[:gerard-majax]

Attachment: mulet-uitests.log

Attached is the log of the current state of running the tests.

Comment 9

[Zac C (:zac)]

(In reply to Alexandre LISSY :gerard-majax from comment #7)
> Until we fix mozrunner/gaiarunner,
> 
> Produce a profile:
> > $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests
> 
> Run mulet:
> > $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html
> 
> Please make sure it has been built with marionette, of course.
> 
> Then start the tests:
> > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log

One addition to the final command line: "--restart" 

The log in comment #8 shows the test failing because this argument is missing.

The tests are designed to work with --restart only although in some cases they will run/pass without it.

Comment 10

[:gerard-majax]

Fine, but:
> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g//mulet-uitests.log

That does start my mulet build, but it does not seems to be passing the app-arg values.

Comment 11

[Zac C (:zac)]

Do you mean that --app-arg worked without restart, but after that it did not?

Comment 12

[:gerard-majax]

(In reply to Zac C (:zac) from comment #11)
> Do you mean that --app-arg worked without restart, but after that it did not?

No, it just cannot properly handle the "-chrome chrome://...".

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

This, however, should work but now I'm getting marionnette errors ...

Comment 13

[:gerard-majax]

Running:

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

Shows an error on http://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#225, i.e., |Services.prefs.getCharPref('b2g.system_startup_url');|

Comment 14

[:gerard-majax]

Proper startup:

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

Comment 15

[Alexandre Poirot [:ochameau]]

Could you provide a patch against gaia ui test runner codebase in order to automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which I imagine is the default and isn't required to be passed) ?
It will have no effect on b2g desktop as the chrome uri is the same, but it will make it work automagically on mulet.

Comment 16

[Zac C (:zac)]

(In reply to Alexandre Poirot (:ochameau) from comment #15)
> Could you provide a patch against gaia ui test runner codebase in order to
> automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which
> I imagine is the default and isn't required to be passed) ?

Yes that's correct, when you use --binary you don't need to use --app=b2gdesktop

Comment 17

[:gerard-majax]

Thanks, now tests are properly starting.

Comment 18

[:gerard-majax]

Attachment: getDisplayedApp error

So now the first test passes but it seems none of the subsequent success, they all fail with this error.

Comment 19

[Zac C (:zac)]

(In reply to Alexandre LISSY :gerard-majax from comment #18)
> Created attachment 8447942 [details]
> getDisplayedApp error
> 
> So now the first test passes but it seems none of the subsequent success,
> they all fail with this error.

OK I think you just need to do `python setup.py develop` to pick up some code changes we made in the last couple of weeks (and before that, wipe or start a fresh virtualenv). 
Locally these test are running and passing well for me. I'll run the Travis test set while I'm out at lunch.

Comment 20

[:gerard-majax]

(In reply to Zac C (:zac) from comment #19)
> (In reply to Alexandre LISSY :gerard-majax from comment #18)
> > Created attachment 8447942 [details]
> > getDisplayedApp error
> > 
> > So now the first test passes but it seems none of the subsequent success,
> > they all fail with this error.
> 
> OK I think you just need to do `python setup.py develop` to pick up some
> code changes we made in the last couple of weeks (and before that, wipe or
> start a fresh virtualenv). 
> Locally these test are running and passing well for me. I'll run the Travis
> test set while I'm out at lunch.

Sorry, but I read in your mind one week ago and I already did all of this:

> (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ which gaiatest
/home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest
> (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ ls -hal /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest
> -rwxr-xr-x 1 alex alex 346 juin  27 13:49 /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest

With a gaia master uptodate of course.

Comment 21

[:gerard-majax]

Attachment: mulet-uitests.log

And yet, after doing another python setup.py develop in the very same virtualenv, tests are much more happy.

I think we can now hack this and start having mulet+ui tests run somewhere.

Comment 22

[:gerard-majax]

Just for the record:

> $ GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g/mulet-uitests.log

Comment 23

[Zac C (:zac)]

Great! It's also running well for me locally, too.

Comment 24

[Alexandre Poirot [:ochameau]]

What is the state of this bug?
Can we run UI tests out of gecko/gaia master?
If yes, we should be able to close this bug and start asking to run this suite on TBPL!

Comment 25

[:gerard-majax]

(In reply to Alexandre Poirot [:ochameau] from comment #24)
> What is the state of this bug?
> Can we run UI tests out of gecko/gaia master?
> If yes, we should be able to close this bug and start asking to run this
> suite on TBPL!

Sure, I had issues the other day when running it, but I'm in the process of doing it right now.
BTW, when doing this I noticed that the gaia-ui-test code living in gaia still does not have the fix from bug 1032799.

Comment 26

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #25)
> (In reply to Alexandre Poirot [:ochameau] from comment #24)
> > What is the state of this bug?
> > Can we run UI tests out of gecko/gaia master?
> > If yes, we should be able to close this bug and start asking to run this
> > suite on TBPL!
> 
> Sure, I had issues the other day when running it, but I'm in the process of
> doing it right now.
> BTW, when doing this I noticed that the gaia-ui-test code living in gaia
> still does not have the fix from bug 1032799.

But according to Jonathan, this is a non issue (cf bug 1032799 comment 10).

Comment 27

[:gerard-majax]

Attachment: Mulet Gaia UI Tests

This is the result for the following call:
> GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --type=b2g --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html gaiatest/tests/tbpl-manifest.ini

SUMMARY
-------
passed: 96
failed: 5
todo: 5

FAILED TESTS
-------
test_sms_add_contact.py test_sms_add_contact.TestSmsAddContact.test_sms_add_contact
test_sms_contact_input_validation.py test_sms_contact_input_validation.TestContactValidation.test_sms_contact_validation
test_sms_contact_match.py test_sms_contact_match.TestContactMatch.test_contact_match
test_mms_add_subject.py test_mms_add_subject.TestMmsAddSubject.test_mms_add_subject
test_inbox_to_settings.py test_inbox_to_settings.TestSettingsFromInbox.test_settings_from_inbox
SUITE-END | took 1362s

Comment 28

[Alexandre Poirot [:ochameau]]

Attachment: patch v1

Here is a patch to start running them on taskcluster.
It looks like I'm almost there:
  https://treeherder.allizom.org/#/jobs?repo=try&revision=377680343ef3
I just get an exception:
  https://taskcluster-artifacts.s3-us-west-2.amazonaws.com/DxMtKT2yT3uqDAClly55Wg/0/public/logs/live_backing.log?AWSAccessKeyId=AKIAIZOQG4W6WG3PH3RQ&Expires=1424698422&Signature=PZ8XnZB7ao%2FVu1r%2F4hB%2FGP8U1JI%3D

Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method Marionette.__del__ of <marionette.marionette.Marionette object at 0x2f5e710>> ignored
11:05:41    ERROR -  Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method GeckoRuntimeRunner.__del__ of <mozrunner.base.browser.GeckoRuntimeRunner object at 0x2f5e9d0>> ignored

Comment 29

[:gerard-majax]

Gip is being killed. No need to continue this.

Comment 30

[Johan Lorenzo [:jlorenzo]]

Agreed.