Status

defect
P1
normal
RESOLVED INVALID
5 years ago
4 years ago

People

(Reporter: gerard-majax, Assigned: gerard-majax)

Tracking

unspecified
x86_64
Linux
Dependency tree / graph

Firefox Tracking Flags

(tracking-b2g:backlog)

Details

(Whiteboard: [systemsfe])

Attachments

(3 attachments, 2 obsolete attachments)

Let's make it work.
Can we get a nightly binary from pvt/releng yet?
It's not ready yet, you have to build it from mozilla-central with 
  ac_add_options --enable-application=b2g/dev
in your mozconfig.

Having tests running and being green is a condition to have nightly builds.
(In reply to Zac C (:zac) from comment #1)
> Can we get a nightly binary from pvt/releng yet?

I think it's in the process of getting built on tbpl. Meanwhile, I'm pulling new gecko master and rebuilding, I can share the result :)
Yes it is already being built in Fig branch over here:
  https://tbpl.mozilla.org/?tree=Fig
And possibly try builds also.
OK thanks Alexandre. With access to pre-built binaries we can run tests locally if/when you need us to.
Depends on: 1023197, 1023200
Until we fix mozrunner/gaiarunner,

Produce a profile:
> $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests

Run mulet:
> $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html

Please make sure it has been built with marionette, of course.

Then start the tests:
> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log
Posted file mulet-uitests.log (obsolete) —
Attached is the log of the current state of running the tests.
(In reply to Alexandre LISSY :gerard-majax from comment #7)
> Until we fix mozrunner/gaiarunner,
> 
> Produce a profile:
> > $ make PROFILE_FOLDER=profile-mulet-uitests profile-mulet-uitests
> 
> Run mulet:
> > $ xvfb-run ./obj-mulet/dist/firefox/firefox -no-remote -jsconsole -profile ../gaia/profile-mulet-uitests/ -chrome chrome://b2g/content/shell.html
> 
> Please make sure it has been built with marionette, of course.
> 
> Then start the tests:
> > $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --address=localhost:2828 --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee mulet-uitests.log

One addition to the final command line: "--restart" 

The log in comment #8 shows the test failing because this argument is missing.

The tests are designed to work with --restart only although in some cases they will run/pass without it.
Fine, but:
> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g//mulet-uitests.log

That does start my mulet build, but it does not seems to be passing the app-arg values.
Do you mean that --app-arg worked without restart, but after that it did not?
(In reply to Zac C (:zac) from comment #11)
> Do you mean that --app-arg worked without restart, but after that it did not?

No, it just cannot properly handle the "-chrome chrome://...".

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

This, however, should work but now I'm getting marionnette errors ...
Running:

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .

Shows an error on http://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#225, i.e., |Services.prefs.getCharPref('b2g.system_startup_url');|
Proper startup:

> $ GAIATEST_SKIP_WARNING=1 gaiatest --app=b2gdesktop --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html .
Could you provide a patch against gaia ui test runner codebase in order to automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which I imagine is the default and isn't required to be passed) ?
It will have no effect on b2g desktop as the chrome uri is the same, but it will make it work automagically on mulet.
(In reply to Alexandre Poirot (:ochameau) from comment #15)
> Could you provide a patch against gaia ui test runner codebase in order to
> automagically pass "-chrome xxx" when running with "--app=b2gdesktop" (which
> I imagine is the default and isn't required to be passed) ?

Yes that's correct, when you use --binary you don't need to use --app=b2gdesktop
Thanks, now tests are properly starting.
So now the first test passes but it seems none of the subsequent success, they all fail with this error.
Flags: needinfo?(zcampbell)
(In reply to Alexandre LISSY :gerard-majax from comment #18)
> Created attachment 8447942 [details]
> getDisplayedApp error
> 
> So now the first test passes but it seems none of the subsequent success,
> they all fail with this error.

OK I think you just need to do `python setup.py develop` to pick up some code changes we made in the last couple of weeks (and before that, wipe or start a fresh virtualenv). 
Locally these test are running and passing well for me. I'll run the Travis test set while I'm out at lunch.
Flags: needinfo?(zcampbell)
(In reply to Zac C (:zac) from comment #19)
> (In reply to Alexandre LISSY :gerard-majax from comment #18)
> > Created attachment 8447942 [details]
> > getDisplayedApp error
> > 
> > So now the first test passes but it seems none of the subsequent success,
> > they all fail with this error.
> 
> OK I think you just need to do `python setup.py develop` to pick up some
> code changes we made in the last couple of weeks (and before that, wipe or
> start a fresh virtualenv). 
> Locally these test are running and passing well for me. I'll run the Travis
> test set while I'm out at lunch.

Sorry, but I read in your mind one week ago and I already did all of this:

> (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ which gaiatest
/home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest
> (gaia-ui-tests_venv)alex@portable-alex:~/codaz/Mozilla/b2g/gaia/tests/python/gaia-ui-tests$ ls -hal /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest
> -rwxr-xr-x 1 alex alex 346 juin  27 13:49 /home/alex/codaz/Mozilla/b2g/gaia-ui-tests_venv/bin/gaiatest

With a gaia master uptodate of course.
Posted file mulet-uitests.log (obsolete) —
And yet, after doing another python setup.py develop in the very same virtualenv, tests are much more happy.

I think we can now hack this and start having mulet+ui tests run somewhere.
Just for the record:

> $ GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-jsconsole" --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --gecko-log=. --testvars=gaiatest/testvars.json --html-output=results.html . 2>&1 | tee ~/codaz/Mozilla/b2g/mulet-uitests.log
Great! It's also running well for me locally, too.
Depends on: 1032799
Depends on: 1035273
Target Milestone: 2.0 S5 (4july) → 2.0 S6 (18july)
Target Milestone: 2.0 S6 (18july) → 2.1 S1 (1aug)
Target Milestone: 2.1 S1 (1aug) → 2.1 S2 (15aug)
What is the state of this bug?
Can we run UI tests out of gecko/gaia master?
If yes, we should be able to close this bug and start asking to run this suite on TBPL!
Flags: needinfo?(lissyx+mozillians)
(In reply to Alexandre Poirot [:ochameau] from comment #24)
> What is the state of this bug?
> Can we run UI tests out of gecko/gaia master?
> If yes, we should be able to close this bug and start asking to run this
> suite on TBPL!

Sure, I had issues the other day when running it, but I'm in the process of doing it right now.
BTW, when doing this I noticed that the gaia-ui-test code living in gaia still does not have the fix from bug 1032799.
Flags: needinfo?(lissyx+mozillians)
(In reply to Alexandre LISSY :gerard-majax from comment #25)
> (In reply to Alexandre Poirot [:ochameau] from comment #24)
> > What is the state of this bug?
> > Can we run UI tests out of gecko/gaia master?
> > If yes, we should be able to close this bug and start asking to run this
> > suite on TBPL!
> 
> Sure, I had issues the other day when running it, but I'm in the process of
> doing it right now.
> BTW, when doing this I noticed that the gaia-ui-test code living in gaia
> still does not have the fix from bug 1032799.

But according to Jonathan, this is a non issue (cf bug 1032799 comment 10).
Posted file Mulet Gaia UI Tests
This is the result for the following call:
> GAIATEST_SKIP_WARNING=1 xvfb-run gaiatest --type=b2g --binary=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox --profile=/home/alex/codaz/Mozilla/b2g/gaia/profile-mulet-uitests/ --app-arg="-chrome" --app-arg="chrome://b2g/content/shell.html" --restart --testvars=gaiatest/testvars.json --html-output=results.html gaiatest/tests/tbpl-manifest.ini

SUMMARY
-------
passed: 96
failed: 5
todo: 5

FAILED TESTS
-------
test_sms_add_contact.py test_sms_add_contact.TestSmsAddContact.test_sms_add_contact
test_sms_contact_input_validation.py test_sms_contact_input_validation.TestContactValidation.test_sms_contact_validation
test_sms_contact_match.py test_sms_contact_match.TestContactMatch.test_contact_match
test_mms_add_subject.py test_mms_add_subject.TestMmsAddSubject.test_mms_add_subject
test_inbox_to_settings.py test_inbox_to_settings.TestSettingsFromInbox.test_settings_from_inbox
SUITE-END | took 1362s
Attachment #8447193 - Attachment is obsolete: true
Attachment #8448015 - Attachment is obsolete: true
Target Milestone: 2.1 S2 (15aug) → 2.1 S3 (29aug)
Target Milestone: 2.1 S3 (29aug) → 2.1 S4 (12sep)
blocking-b2g: --- → backlog
Priority: -- → P1
Target Milestone: 2.1 S4 (12sep) → ---
QA Whiteboard: [fxosqa-auto-backlog-]
Posted patch patch v1Splinter Review
Here is a patch to start running them on taskcluster.
It looks like I'm almost there:
  https://treeherder.allizom.org/#/jobs?repo=try&revision=377680343ef3
I just get an exception:
  https://taskcluster-artifacts.s3-us-west-2.amazonaws.com/DxMtKT2yT3uqDAClly55Wg/0/public/logs/live_backing.log?AWSAccessKeyId=AKIAIZOQG4W6WG3PH3RQ&Expires=1424698422&Signature=PZ8XnZB7ao%2FVu1r%2F4hB%2FGP8U1JI%3D

Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method Marionette.__del__ of <marionette.marionette.Marionette object at 0x2f5e710>> ignored
11:05:41    ERROR -  Exception AttributeError: AttributeError("'ProcessHandler' object has no attribute 'proc'",) in <bound method GeckoRuntimeRunner.__del__ of <mozrunner.base.browser.GeckoRuntimeRunner object at 0x2f5e9d0>> ignored
blocking-b2g: backlog → ---
Gip is being killed. No need to continue this.
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(jlorenzo)
Resolution: --- → INVALID
Agreed.
Flags: needinfo?(jlorenzo)
You need to log in before you can comment on or make changes to this bug.