Closed Bug 1173933 Opened 9 years ago Closed 9 years ago

We should allow for more than one tab

Categories

(Testing :: Firefox UI Tests, defect)

38 Branch
defect
Not set
normal

Tracking

(firefox38 fixed, firefox39 fixed, firefox40 fixed, firefox41 fixed, firefox-esr38 fixed)

RESOLVED FIXED
mozilla41
Tracking Status
firefox38 --- fixed
firefox39 --- fixed
firefox40 --- fixed
firefox41 --- fixed
firefox-esr38 --- fixed

People

(Reporter: armenzg, Assigned: chmanchester)

References

Details

Attachments

(2 files)

I believe the reason is because after the update we get the pocket window.
This update goes from 38.0 to 38.0.5

See attached log to see the real issue.
Also another issue is that we can't see this error in the mozharness log [1]
There is a bug in the mozharness code that looks for gecko.txt instead of gecko.log.

You can reproduce this failure with this command:
 python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py

You cannot reproduce the issue for the 39 betas (since there is no pocket tab):
 python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0b3-candidates/build1/mac/en-US/Firefox%2039.0b3.dmg --cfg developer_config.py --run-tests

[1]
 1:59.98 TEST_END: MainThread ERROR, expected PASS
 Traceback (most recent call last):
  File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 326, in run
    self.tearDown()
  File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_tests/update/fallback/test_fallback_update.py", line 17, in tearDown
    UpdateTestCase.tearDown(self)
  File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_harness/testcases/update.py", line 235, in tearDown
    FirefoxTestCase.tearDown(self)
  File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_harness/testcases/base.py", line 75, in tearDown
    (self._start_handle_count, win_count))
 AssertionError: A test must not leak window handles. This test started the browser with 1 open top level browsing contexts, but ended with 2.

[2]
15:15:10     INFO - Current package versions:
15:15:10     INFO -   manifestparser == 1.1
15:15:10     INFO -   wheel == 0.24.0
15:15:10     INFO -   mozhttpd == 0.7
15:15:10     INFO -   mozprofile == 0.23
15:15:10     INFO -   marionette-transport == 0.4
15:15:10     INFO -   marionette-client == 0.12
15:15:10     INFO -   mozcrash == 0.14
15:15:10     INFO -   mozfile == 1.1
15:15:10     INFO -   firefox-ui-tests == 0.2
15:15:10     INFO -   mozlog == 2.10
15:15:10     INFO -   moznetwork == 0.24
15:15:10     INFO -   mozInstall == 1.12
15:15:10     INFO -   mozdevice == 0.44
15:15:10     INFO -   marionette-driver == 0.7
15:15:10     INFO -   blessings == 1.5.1
15:15:10     INFO -   mozversion == 1.1
15:15:10     INFO -   mozprocess == 0.21
15:15:10     INFO -   mozrunner == 6.6
15:15:10     INFO -   moztest == 0.7
15:15:10     INFO -   mozinfo == 0.8
...
15:16:41     INFO - Copy/paste: /Users/armenzg/repos/hg/mozharness/build/venv/bin/firefox-ui-update --installer /Users/armenzg/repos/hg/mozharness/build/Firefox%2038.0.dmg --gecko-log=/Users/armenzg/repos/hg/mozharness/build/gecko.log
...
15:16:41     INFO -   0:00.01 LOG: MainThread INFO Installing application "/Users/armenzg/repos/hg/mozharness/build/Firefox%2038.0.dmg" to "/var/folders/wl/l70t_q8n6gg64vr7bfzjrqg80000gn/T/tmpuobWcN"
15:17:05     INFO -   0:23.09 LOG: MainThread INFO Creating a copy of the application at "/var/folders/wl/l70t_q8n6gg64vr7bfzjrqg80000gn/T/tmpQSUcBt.binary-update-tests".
15:17:09     INFO -   0:27.05 LOG: MainThread INFO starting httpd
15:17:09     INFO -   0:27.06 LOG: MainThread INFO running httpd on http://127.0.0.1:52793/
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_buildid: 20150508094354
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_changeset: 4c4dc6640c7e
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_display_name: Firefox
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_name: Firefox
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_remotingname: firefox
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_repository: https://hg.mozilla.org/releases/mozilla-release
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_vendor: Mozilla
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO application_version: 38.0
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO platform_buildid: 20150508094354
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO platform_changeset: 4c4dc6640c7e
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO platform_repository: https://hg.mozilla.org/releases/mozilla-release
15:17:09     INFO -   0:27.06 LOG: MainThread mozversion INFO platform_version: 38.0
15:17:09     INFO -   0:27.06 SUITE_START: MainThread 1
15:17:09     INFO -   0:27.06 TEST_START: MainThread test_direct_update.py TestDirectUpdate.test_update
15:17:16     INFO -   0:34.53 TEST_END: MainThread ERROR, expected PASS
15:17:16     INFO -  Traceback (most recent call last):
^@15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 296, in run
15:17:16     INFO -      testMethod()
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_tests/update/direct/test_direct_update.py", line 26, in test_update
15:17:16     INFO -      self._test_update()
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_tests/update/direct/test_direct_update.py", line 20, in _test_update
15:17:16     INFO -      self.download_and_apply_available_update(force_fallback=False)
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_ui_harness/testcases/update.py", line 147, in download_and_apply_available_update
15:17:16     INFO -      about_window = self.browser.open_about_window()
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_puppeteer/ui/windows.py", line 576, in open_about_window
15:17:16     INFO -      return BaseWindow.open_window(self, callback, AboutWindow)
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/firefox_puppeteer/ui/windows.py", line 352, in open_window
15:17:16     INFO -      Wait(self.marionette).until(lambda _: window.focused == expect_focus)
15:17:16     INFO -    File "/Users/armenzg/repos/hg/mozharness/build/venv/lib/python2.7/site-packages/marionette_driver/wait.py", line 143, in until
15:17:16     INFO -      cause=last_exc)
15:17:16     INFO -  TimeoutException: TimeoutException: Timed out after 5.1 seconds
15:17:16     INFO -   0:34.56 LOG: MainThread INFO Update test results:
15:17:16     INFO -  [{'build_post': None,
15:17:16     INFO -    'build_pre': {'buildid': u'20150508094354',
15:17:16     INFO -                  'channel': u'release',
15:17:16     INFO -                  'disabled_addons': None,
15:17:16     INFO -                  'locale': u'en-US',
15:17:16     INFO -                  'mar_channels': set([u'firefox-mozilla-release']),
15:17:16     INFO -                  'url_aus': u'https://aus4.mozilla.org/update/3/Firefox/38.0/20150508094354/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2014.3.0/default/default/update.xml?force=1',
15:17:16     INFO -                  'user_agent': u'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0',
15:17:16     INFO -                  'version': u'38.0'},
15:17:16     INFO -    'fallback': False,
15:17:16     INFO -    'patch': {},
15:17:16     INFO -    'success': False}]
15:17:16     INFO -   0:34.63 LOG: MainThread INFO
15:17:16     INFO -  SUMMARY
15:17:16     INFO -  -------
15:17:16     INFO -   0:34.63 LOG: MainThread INFO passed: 0
15:17:16     INFO -   0:34.63 LOG: MainThread INFO failed: 1
15:17:16     INFO -   0:34.64 LOG: MainThread INFO todo: 0
15:17:16     INFO -   0:34.64 LOG: MainThread INFO
15:17:16     INFO -  FAILED TESTS
15:17:16     INFO -  -------
15:17:16     INFO -   0:34.64 LOG: MainThread INFO test_direct_update.py test_direct_update.TestDirectUpdate.test_update
15:17:16     INFO -   0:34.69 SUITE_END: MainThread
15:17:16     INFO -  Summary
15:17:16     INFO -  =======
15:17:16     INFO -  Ran 1 tests
15:17:16     INFO -  Expected results: 0
15:17:16     INFO -  Unexpected results: 1 (ERROR: 1)
15:17:16     INFO -  Unexpected Results
As discussed on irc, what we'd like to do is close open tabs in teardown so we might be able to add assertions later that these promotional tabs are appearing as expected.
Assignee: nobody → cmanchester
That's correct. Something we have to do in the future is to also check for the what's new page if necessary. We cannot limit Firefox to only use a single tab.
Attached file PR with fix
Attachment #8621237 - Flags: review?(armenzg)
Comment on attachment 8621237 [details] [review]
PR with fix

I've tested that this works.

This can be reproduced like this:
1) Run with --no-run-tests
 python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --no-run-tests --cfg developer_config.py

2) Apply the patch on build/firefox_ui_tests

3) Re-create venv & run-tests
 rm -rf build/venv && python scripts/firefox_ui_updates.py --installer-path `pwd`/build/Firefox%2038.0.dmg --create-virtualenv --run-tests --cfg developer_config.py
Attachment #8621237 - Flags: review?(armenzg) → review+
We need the change on all firefox_ui_test branches.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 41
Version: Firefox 41 → Firefox 38
Product: Mozilla QA → Testing
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: