52 bytes, text/x-github-pull-request
|Details | Review | Splinter Review|
52 bytes, text/x-github-pull-request
|Details | Review | Splinter Review|
1.08 KB, patch
|Details | Diff | Splinter Review|
The test fails currently on Windows only with the following failure: Traceback (most recent call last): File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_client-0.12-py2.7.egg\marionette\marionette_test.py", line 296, in run testMethod() File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\decorators.py", line 15, in wrapper return target(self, *args, **kwargs) File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\decorators.py", line 23, in wrapper return target(self, *args, **kwargs) File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\test_ssl_status_after_restart.py", line 61, in test_ssl_status_after_restart self.verify_certificate_status(item) File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\test_ssl_status_after_restart.py", line 74, in verify_certificate_status Wait(self.marionette).until(lambda _: self.identity_popup.is_open) File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_driver-0.5-py2.7.egg\marionette_driver\wait.py", line 143, in until cause=last_exc) TimeoutException: TimeoutException: Timed out after 5.0 seconds The reason here is that the appropriate tab is not selected before. This happens because the test does not wait until the session has been restored. As long as we cannot wait for the appropriate sessionrestored event, we should add a sleep. Maybe the best location for that is directly in the FirefoxTestCase.restart() method. That way each test will benefit from it.
This failure seems to only happen for the Windows XP node in SCL3. All other Windows machines are working fine. I wonder if it is something related to this machine then. I will see if I can find a local XP machine for tests.
OS: Unspecified → Windows XP
Created attachment 8611467 [details] [review] skip patch for Windows XP This is the skip patch for Windows XP only. Maybe someone will have the time to investigate. I don't have it at the moment. I will land this change once Travis CI is passing.
The test has been disabled for Windows XP now: https://github.com/mozilla/firefox-ui-tests/commit/ad6817abb55b67f106ed20cd40a00374037fde08 (master) Backports I will do soon once all other failures are fixed, which also need a backport.
status-firefox38: --- → affected
status-firefox39: --- → affected
status-firefox40: --- → affected
status-firefox41: --- → disabled
status-firefox-esr38: --- → affected
The patch as landed doesn't work because the machines report: version: 5.1.2600 os: win service_pack: Service Pack 3 bits: 32 processor: x86 That means os_version == "5.1" does not match. I feel this is a bug in handling OS versions in the manifest parser. It should not take the build number into account. I will check this.
The problem is that we haven't had a new release of mozinfo since Oct 2013! Features like os_version are not available via the packages on PyPI. I'm going to release version 0.8 via bug 1169168 to unblock us.
I also pushed a version bump for mozinfo to 0.8: https://github.com/mozilla/firefox-ui-tests/commit/6aa74d806b373ba29cee83d90e0ebe501192c48b If that works I will have to backport both patches down to esr38.
Having our tests for Nightly run in production now, I can also see those timeouts on other versions of Windows like Vista, 7, and 8. So I think we should skip it for all Windows, and check soon what's going on.
OS: Windows XP → Windows
Landed the skip all Windows change via: https://github.com/mozilla/firefox-ui-tests/commit/ca50ad13e30eb272511c69647f8d4ea58cd461d9
Backports for mozilla-aurora: https://github.com/mozilla/firefox-ui-tests/commit/7254d99527e7cde7adf4779ebf604923d513ac9e https://github.com/mozilla/firefox-ui-tests/commit/05664d9356ff01dd2adbf1130a1be23955e98e3f https://github.com/mozilla/firefox-ui-tests/commit/4a3c39c8339ee6186bd9c771783a6512f7e1a96f Backports for mozilla-beta, mozilla-release, and mozilla-esr38: https://github.com/mozilla/firefox-ui-tests/commit/be91d8d54d09a953cc1c5b7c3d492a12c5a94f1b https://github.com/mozilla/firefox-ui-tests/commit/1c93b211bbd8e05a23458dac487e3770612335f2 https://github.com/mozilla/firefox-ui-tests/commit/455191588e74344687899f6a7c8cf1c4a78ac67f
status-firefox38: affected → disabled
status-firefox39: affected → disabled
status-firefox40: affected → disabled
status-firefox-esr38: affected → disabled
No longer blocks: 1165215
While working on bug 1128968 I noticed that we missed to retrieve the elements for the identity popup again after the restart. That's why it was failing: https://github.com/mozilla/firefox-ui-tests/commit/2c38349a1494aa6b43f82789785638f467ec031b#diff-c2229f1aac10fba9af79defdcaf44a78R59 I'm not sure why it's working on some platforms but not all. I would wait with further investigation until bug 1177417 has been solved.
Depends on: 1177417
No longer depends on: 1177417
I did a couple of local runs for this test on Windows and I was not able to see this failure anymore. That means I would like to try to unskip this test. A simple backout will not work due to the remote->functional migration. So I will come up with a new patch.
Created attachment 8705079 [details] [review] Github PR (unskip test)
Attachment #8705079 - Flags: review?(mjzffr)
Attachment #8705079 - Flags: review?(mjzffr) → review+
Merged PR to mozilla-central: https://github.com/mozilla/firefox-ui-tests/commit/ca7e96357b92678e65ac75c62fbbf400ff6c5f3b If we don't have any failures I would like to also get it enabled for mozilla-aurora which will be the next ESR release.
status-firefox38: disabled → ---
status-firefox39: disabled → ---
status-firefox40: disabled → ---
status-firefox41: disabled → ---
status-firefox43: --- → disabled
status-firefox44: --- → disabled
status-firefox45: --- → disabled
status-firefox46: --- → fixed
Target Milestone: --- → mozilla46
So all Windows platforms seem to work except for Win XP when we see this failure again now: https://treeherder.mozilla.org/logviewer.html#?job_id=3035636&repo=mozilla-central I will leave this running for the weekend and check if we constantly fail.
This failure always reproduces on WinXP via the nodes mm-win-xp-32-3 and mm-win-xp-32-4. The permanent issue with OS X 10.10 is bug 1198611. It was working fine since lately. I will check both bugs in more detail tomorrow.
The failure on WinXP was not visible the last two days. Maybe something has been changed here. So it's not permanent. I would leave this test enabled on mozilla-central for now and also enable it in aurora. That way we might get a better idea what's going on if it happens again. Backported backout patch to mozilla-aurora: https://github.com/mozilla/firefox-ui-tests/commit/e570df4679256c5f4972680937ef89f3db449ea7
status-firefox45: disabled → affected
status-firefox46: fixed → affected
OS: Windows → Windows XP
Summary: Failure in test_ssl_status_after_restart.py because test doesn't wait after restart for session restored → [WinXP] TEST-UNEXPECTED-ERROR | test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.1 seconds
Nowadays this is a permanent failure on both mozilla-central and mozilla-aurora. So we should get this turned off again for WinXP only. I don't think it makes that much sense right now for investigation.
Created attachment 8711633 [details] [review] Github PR (Re-skip test but on WinXP only) This is similar to attachment 8611467 [details] [review] but with an updated path.
Created attachment 8711635 [details] [diff] [review] Disable patch (v1) [mozilla-central] Adapted patch for mozilla-central on hg.mo.
Attachment #8711635 - Flags: review?(mjzffr)
Attachment #8711633 - Flags: review?(mjzffr) → review+
Attachment #8711635 - Flags: review?(mjzffr) → review+
I landed the Github patch across all affected branches: https://github.com/mozilla/firefox-ui-tests/commit/0c64b99f0c69c216b3720887609e6047cdd5b74f (central) https://github.com/mozilla/firefox-ui-tests/commit/e124685d218196b8673f5f03a836c5bd1af87d22 (aurora) https://github.com/mozilla/firefox-ui-tests/commit/764b070502acfccdf110e67762aff7624a54f6da (beta) I will also land it on inbound now. I leave the status flags for proper landing on hg.mo.
status-firefox47: --- → affected
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
status-firefox47: affected → fixed
Resolution: --- → FIXED
Attachment 8711635 [details] [diff] is a testonly change and does not affect the product. It constantly fails for us on WinXP, so we would like to have it skipped down to the upcoming 45.0 ESR.
status-firefox45: affected → fixed
status-firefox46: affected → fixed
Actually I will mark this bug as wontfix given that it only affects a platform which we can hopefully get rid of in the next months. That's why I don't want to put in any other work.
Resolution: FIXED → WONTFIX
You need to log in before you can comment on or make changes to this bug.