1256570 - Add regression test for bug 1178722 "Select with multiple selection - options visibility not respected in popup"

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Tristan]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36




Expected results:

Bug 1178722 has been fixed, but requires a regression test. :Margaret Leibovic has recommended a Chrome Mochitest. For more information, see https://bugzilla.mozilla.org/show_bug.cgi?id=1178722#c12.

Comment 1

[Tristan]

> I agree the test framework isn't very approachable, but you should be able
> to get something basic up and running by copy/pasting from existing tests.
> Can you file a follow-up bug to track this work? You don't need to solve it
> immediately, but having a bug on file will be good for remembering that this
> issue exists.

I can get the Chrome Mochitests to run on my phone, and I think I understand the basics of how the tests are supposed to be written. 

Where I'm struggling is reading and interacting with the popup prompt. Are there any specific docs/examples that you'd recommend I look at? I could work it out with lots of trial and error, but it takes about 4 minutes to start mochitest. Is that normal? 

Thank you :)


From terminal...


`user@host:~/w/mozilla-central$ ./mach mochitest -f chrome mobile/android/tests/browser/chrome/test_hidden_select_option.html
Elapsed: 1.88s; From _tests: Kept 20467 existing; Added/updated 0; Removed 0 files and 0 directories.

######
### Now running mochitest-chrome.
######

 0:03.98 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
 0:04.41 LOG: MainThread INFO Checking for orphan ssltunnel processes...
 0:04.50 LOG: MainThread INFO Checking for orphan xpcshell processes...
 0:04.60 SUITE_START: MainThread 1
 0:14.56 LOG: MainThread INFO pushing /home/me/w/mozilla-central/objdir-frontend/_tests/testing/mochitest/chrome to /storage/sdcard/tests/chrome on device...
 0:42.67 LOG: MainThread WARNING runtests.py | Failed to copy /home/me/w/mozilla-central/objdir-frontend/dist/plugins to profile
pk12util: PKCS12 IMPORT SUCCESSFUL
 1:35.66 LOG: MainThread INFO MochitestServer : launching [u'/home/me/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/xpcshell', '-g', '/home/me/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64', '-v', '170', '-f', '/home/me/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpDpLTPz.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.0.18'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/me/w/mozilla-central/objdir-frontend/_tests/testing/mochitest/server.js']
 1:35.66 LOG: MainThread INFO runtests.py | Server pid: 17546
 1:35.68 LOG: MainThread INFO runtests.py | Websocket server pid: 17549
 1:35.73 LOG: MainThread INFO runtests.py | SSL tunnel pid: 17552
 2:27.76 LOG: MainThread INFO runtests.py | Running tests: start.

INFO | automation.py | Application pid: 2331
 3:05.76 LOG: None INFO SimpleTest START
 3:05.81 TEST_START: None mobile/android/tests/browser/chrome/test_hidden_select_option.html
`

Comment 2

[:Margaret Leibovic]

(In reply to Tristan from comment #1)
> > I agree the test framework isn't very approachable, but you should be able
> > to get something basic up and running by copy/pasting from existing tests.
> > Can you file a follow-up bug to track this work? You don't need to solve it
> > immediately, but having a bug on file will be good for remembering that this
> > issue exists.
> 
> I can get the Chrome Mochitests to run on my phone, and I think I understand
> the basics of how the tests are supposed to be written. 
> 
> Where I'm struggling is reading and interacting with the popup prompt. Are
> there any specific docs/examples that you'd recommend I look at?

This is tricky, since this would be specific to writing this test. If you're going to use a Chrome Mochitest, you'll only have access to JS. In this case, I think you should try unit-testing the SelectHelper.js code. You should be able to get at the SelectHelper object directly by calling something like this:

let win = Services.wm.getMostRecentWindow("navigator:browser");
let SelectHelper = win.SelectHelper;

Ideally we would test the whole flow and verify what's shown in the prompt, but our Android UI tests can be finicky, and I think that we'll have better results (i.e. fewer intermittent failures) if we just try testing the JS logic.

You may need to refactor the logic a bit to make it more unit-test-able, but I think that would be fine.

If you do want to try testing the Android UI in combination with the JS, you'll have to use a JavascriptTest, which is a variant of our robocop tests. testTrackingProtection is a good example of one of these tests:
http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/testTrackingProtection.js
http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/tests/testTrackingProtection.java

> I could work it out with lots of trial and error, but it takes about 4 minutes to
> start mochitest. Is that normal? 

It can be slow, but that's pretty bad... are you running these on a device or in the emulator?

As a pro tip, you can use WebIDE and the remote console to do trial and error testing outside of your test code:
https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Debugging_Firefox_for_Android_with_WebIDE


PS - I gave you permissions to confirm a bug, so now new bugs you file will be status NEW.

Comment 3

[Tristan]

regression tests added to 1178722