1309141 - AssertionError in test_browser_window test suite on hard shutdown for test_quit_firefox_shortcut

Status: RESOLVED INCOMPLETE

(Remote Protocol :: Marionette, defect)

Description

[Sanyam Khurana [:CuriousLearner]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36

Steps to reproduce:

While working on bug 1298803 to Add test to quit Firefox via a shortcut, the following test was added in `testing/firefox-ui/tests/functional/keyboard_shortcuts/test_browser_window.py`

> def test_quit_firefox_shortcut(self):
> 
>         def quit_via_shortcut_callback():
>             if self.platform == 'win':
>                 key = 'quitApplicationCmdWin2.accesskey'
>             else:
>                 key = 'quitApplicationCmdUnix.key'
> 
>             self.browser.send_shortcut(self.browser.get_entity(key))
> 
>         self.marionette.quit(callback=quit_via_shortcut_callback)
>         self.marionette.start_session()

If you pass `in_app=True` in the call to quit method, then all test cases pass; but they should also pass a forced shutdown is done.


Actual results:

The 2nd test case to `test_search_field` fails with an `AssertionError`. Here is the full traceback:

$ ./mach firefox-ui-functional /Users/sanyam/personal/mozilla/mozilla-central/testing/firefox-ui/tests/functional/keyboard_shortcuts/test_browser_window.py
 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/sanyam/personal/mozilla/mozilla-central"
 0:00.02 LOG: MainThread INFO Application command: /Users/sanyam/personal/mozilla/mozilla-central/obj-x86_64-apple-darwin15.6.0/dist/NightlyDebug.app/Contents/MacOS/firefox -no-remote -marionette -foreground -profile /var/folders/16/zkkbtd3n1td1fb0z6lxhk2pw0000gn/T/tmpsizgas.mozrunner
 0:34.29 LOG: MainThread INFO Initial Profile Destination is "/var/folders/16/zkkbtd3n1td1fb0z6lxhk2pw0000gn/T/tmpsizgas.mozrunner"
 0:34.29 LOG: MainThread INFO starting httpd
 0:34.30 LOG: MainThread INFO running httpd on http://127.0.0.1:63270/
 0:34.30 LOG: MainThread mozversion INFO application_buildid: 20161010185031
 0:34.30 LOG: MainThread mozversion INFO application_changeset: 7be6b348c431d69f96f0765af3a0c0a0fe56d4bf
 0:34.30 LOG: MainThread mozversion INFO application_display_name: Nightly
 0:34.30 LOG: MainThread mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 0:34.30 LOG: MainThread mozversion INFO application_name: Firefox
 0:34.30 LOG: MainThread mozversion INFO application_remotingname: firefox
 0:34.30 LOG: MainThread mozversion INFO application_vendor: Mozilla
 0:34.30 LOG: MainThread mozversion INFO application_version: 52.0a1
 0:34.30 LOG: MainThread mozversion INFO platform_buildid: 20161010185031
 0:34.30 LOG: MainThread mozversion INFO platform_changeset: 7be6b348c431d69f96f0765af3a0c0a0fe56d4bf
 0:34.30 LOG: MainThread mozversion INFO platform_version: 52.0a1
 0:34.31 LOG: MainThread INFO running with e10s: True
 0:34.31 SUITE_START: MainThread 1
 0:34.31 TEST_START: MainThread test_browser_window.py TestBrowserWindowShortcuts.test_addons_manager
 1:29.35 TEST_END: MainThread PASS
 1:29.35 TEST_START: MainThread test_browser_window.py TestBrowserWindowShortcuts.test_quit_firefox_shortcut
 1:43.49 LOG: MainThread INFO Application command: /Users/sanyam/personal/mozilla/mozilla-central/obj-x86_64-apple-darwin15.6.0/dist/NightlyDebug.app/Contents/MacOS/firefox -no-remote -marionette -foreground -profile /var/folders/16/zkkbtd3n1td1fb0z6lxhk2pw0000gn/T/tmpzEAAwo.mozrunner
 2:15.82 TEST_END: MainThread PASS
 2:15.82 TEST_START: MainThread test_browser_window.py TestBrowserWindowShortcuts.test_search_field
 2:21.22 TEST_END: MainThread FAIL, expected PASS
Traceback (most recent call last):
  File "/Users/sanyam/personal/mozilla/mozilla-central/testing/marionette/harness/marionette/marionette_test/testcases.py", line 162, in run
    testMethod()
  File "/Users/sanyam/personal/mozilla/mozilla-central/testing/firefox-ui/tests/functional/keyboard_shortcuts/test_browser_window.py", line 37, in test_search_field
    self.assertNotEqual(current_name, "input")
AssertionError: u'input' == 'input'
 2:24.14 LOG: MainThread INFO 
SUMMARY
-------
 2:24.14 LOG: MainThread INFO passed: 2
 2:24.14 LOG: MainThread INFO failed: 1
 2:24.14 LOG: MainThread INFO todo: 0
 2:24.14 LOG: MainThread INFO 
FAILED TESTS
-------
 2:24.14 LOG: MainThread INFO test_browser_window.py test_browser_window.TestBrowserWindowShortcuts.test_search_field
 2:24.14 LOG: MainThread INFO mode: e10s
 2:24.14 SUITE_END: MainThread 
Summary
=======

Ran 3 tests
Expected results: 2
Unexpected results: 1 (FAIL: 1)

Unexpected Results
==================

FAIL test_browser_window.py TestBrowserWindowShortcuts.test_search_field


Expected results:

All tests should've passed even when in_app parameter is not passed as True.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 02/17 - 02/21)]

Sanyam, it would be good if you can create a minimized testcase, which means that you only take those code pieces which actually cause this behavior and put those in a single file. Remove every other code. Once done, you can upload it to this bug. Thanks.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 02/17 - 02/21)]

If you can reproduce this issue please reopen the bug.