1167179 - [WinXP] TEST-UNEXPECTED-ERROR | test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.1 seconds

Status: RESOLVED WONTFIX

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The test fails currently on Windows only with the following failure:

Traceback (most recent call last):
  File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_client-0.12-py2.7.egg\marionette\marionette_test.py", line 296, in run
    testMethod()
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\decorators.py", line 15, in wrapper
    return target(self, *args, **kwargs)
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\decorators.py", line 23, in wrapper
    return target(self, *args, **kwargs)
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\test_ssl_status_after_restart.py", line 61, in test_ssl_status_after_restart
    self.verify_certificate_status(item)
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\test_ssl_status_after_restart.py", line 74, in verify_certificate_status
    Wait(self.marionette).until(lambda _: self.identity_popup.is_open)
  File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_driver-0.5-py2.7.egg\marionette_driver\wait.py", line 143, in until
    cause=last_exc)

TimeoutException: TimeoutException: Timed out after 5.0 seconds

The reason here is that the appropriate tab is not selected before.  This happens because the test does not wait until the session has been restored.

As long as we cannot wait for the appropriate sessionrestored event, we should add a sleep. Maybe the best location for that is directly in the FirefoxTestCase.restart() method. That way each test will benefit from it.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This failure seems to only happen for the Windows XP node in SCL3. All other Windows machines are working fine. I wonder if it is something related to this machine then. I will see if I can find a local XP machine for tests.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: skip patch for Windows XP

This is the skip patch for Windows XP only. Maybe someone will have the time to investigate. I don't have it at the moment. I will land this change once Travis CI is passing.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The test has been disabled for Windows XP now:
https://github.com/mozilla/firefox-ui-tests/commit/ad6817abb55b67f106ed20cd40a00374037fde08 (master)

Backports I will do soon once all other failures are fixed, which also need a backport.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The patch as landed doesn't work because the machines report:

version: 5.1.2600
os: win
service_pack: Service Pack 3
bits: 32
processor: x86

That means os_version == "5.1" does not match. I feel this is a bug in handling OS versions in the manifest parser. It should not take the build number into account. I will check this.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The problem is that we haven't had a new release of mozinfo since Oct 2013! Features like os_version are not available via the packages on PyPI. I'm going to release version 0.8 via bug 1169168 to unblock us.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I also pushed a version bump for mozinfo to 0.8:
https://github.com/mozilla/firefox-ui-tests/commit/6aa74d806b373ba29cee83d90e0ebe501192c48b

If that works I will have to backport both patches down to esr38.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Having our tests for Nightly run in production now, I can also see those timeouts on other versions of Windows like Vista, 7, and 8. So I think we should skip it for all Windows, and check soon what's going on.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed the skip all Windows change via:
https://github.com/mozilla/firefox-ui-tests/commit/ca50ad13e30eb272511c69647f8d4ea58cd461d9

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Backports for mozilla-aurora:

https://github.com/mozilla/firefox-ui-tests/commit/7254d99527e7cde7adf4779ebf604923d513ac9e
https://github.com/mozilla/firefox-ui-tests/commit/05664d9356ff01dd2adbf1130a1be23955e98e3f
https://github.com/mozilla/firefox-ui-tests/commit/4a3c39c8339ee6186bd9c771783a6512f7e1a96f

Backports for mozilla-beta, mozilla-release, and mozilla-esr38:
https://github.com/mozilla/firefox-ui-tests/commit/be91d8d54d09a953cc1c5b7c3d492a12c5a94f1b
https://github.com/mozilla/firefox-ui-tests/commit/1c93b211bbd8e05a23458dac487e3770612335f2
https://github.com/mozilla/firefox-ui-tests/commit/455191588e74344687899f6a7c8cf1c4a78ac67f

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

While working on bug 1128968 I noticed that we missed to retrieve the elements for the identity popup again after the restart. That's why it was failing:

https://github.com/mozilla/firefox-ui-tests/commit/2c38349a1494aa6b43f82789785638f467ec031b#diff-c2229f1aac10fba9af79defdcaf44a78R59

I'm not sure why it's working on some platforms but not all. I would wait with further investigation until bug 1177417 has been solved.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I did a couple of local runs for this test on Windows and I was not able to see this failure anymore. That means I would like to try to unskip this test. A simple backout will not work due to the remote->functional migration. So I will come up with a new patch.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (unskip test)

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Merged PR to mozilla-central:
https://github.com/mozilla/firefox-ui-tests/commit/ca7e96357b92678e65ac75c62fbbf400ff6c5f3b

If we don't have any failures I would like to also get it enabled for mozilla-aurora which will be the next ESR release.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So all Windows platforms seem to work except for Win XP when we see this failure again now:

https://treeherder.mozilla.org/logviewer.html#?job_id=3035636&repo=mozilla-central

I will leave this running for the weekend and check if we constantly fail.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This failure always reproduces on WinXP via the nodes mm-win-xp-32-3 and mm-win-xp-32-4. The permanent issue with OS X 10.10 is bug 1198611. It was working fine since lately. I will check both bugs in more detail tomorrow.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The failure on WinXP was not visible the last two days. Maybe something has been changed here. So it's not permanent. I would leave this test enabled on mozilla-central for now and also enable it in aurora. That way we might get a better idea what's going on if it happens again.

Backported backout patch to mozilla-aurora:
https://github.com/mozilla/firefox-ui-tests/commit/e570df4679256c5f4972680937ef89f3db449ea7

Comment 17

[Intermittent Failures Robot]

15 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 11
* mozilla-central: 4

Platform breakdown:
* windowsxp: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1167179&startday=2016-01-18&endday=2016-01-24&tree=all

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Nowadays this is a permanent failure on both mozilla-central and mozilla-aurora. So we should get this turned off again for WinXP only. I don't think it makes that much sense right now for investigation.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (Re-skip test but on WinXP only)

This is similar to attachment 8611467 [details] [review] but with an updated path.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Disable patch (v1) [mozilla-central]

Adapted patch for mozilla-central on hg.mo.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I landed the Github patch across all affected branches:

https://github.com/mozilla/firefox-ui-tests/commit/0c64b99f0c69c216b3720887609e6047cdd5b74f (central)
https://github.com/mozilla/firefox-ui-tests/commit/e124685d218196b8673f5f03a836c5bd1af87d22 (aurora)
https://github.com/mozilla/firefox-ui-tests/commit/764b070502acfccdf110e67762aff7624a54f6da (beta)

I will also land it on inbound now. I leave the status flags for proper landing on hg.mo.

Comment 22

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/48292bc3676d

Comment 23

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/48292bc3676d

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment 8711635 [details] [diff] is a testonly change and does not affect the product. It constantly fails for us on WinXP, so we would like to have it skipped down to the upcoming 45.0 ESR.

Comment 25

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/560a24bbebce
 https://hg.mozilla.org/releases/mozilla-beta/rev/07bd7f4a0a1b

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually I will mark this bug as wontfix given that it only affects a platform which we can hopefully get rid of in the next months. That's why I don't want to put in any other work.