Closed Bug 1167179 Opened 5 years ago Closed 4 years ago

[WinXP] TEST-UNEXPECTED-ERROR | TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.1 seconds


(Testing :: Firefox UI Tests, defect)

Windows XP
Not set


(firefox43 disabled, firefox44 disabled, firefox45 fixed, firefox46 fixed, firefox47 fixed, firefox-esr38 disabled)

Tracking Status
firefox43 --- disabled
firefox44 --- disabled
firefox45 --- fixed
firefox46 --- fixed
firefox47 --- fixed
firefox-esr38 --- disabled


(Reporter: whimboo, Assigned: whimboo)



(Keywords: intermittent-failure)


(3 files, 1 obsolete file)

The test fails currently on Windows only with the following failure:

Traceback (most recent call last):
  File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_client-0.12-py2.7.egg\marionette\", line 296, in run
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\", line 15, in wrapper
    return target(self, *args, **kwargs)
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_harness\", line 23, in wrapper
    return target(self, *args, **kwargs)
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\", line 61, in test_ssl_status_after_restart
  File "c:\jenkins\workspace\mozilla-central_remote\firefox-ui-tests\firefox_ui_tests\remote\security\", line 74, in verify_certificate_status
    Wait(self.marionette).until(lambda _: self.identity_popup.is_open)
  File "c:\jenkins\workspace\mozilla-central_remote\venv\lib\site-packages\marionette_driver-0.5-py2.7.egg\marionette_driver\", line 143, in until

TimeoutException: TimeoutException: Timed out after 5.0 seconds

The reason here is that the appropriate tab is not selected before.  This happens because the test does not wait until the session has been restored.

As long as we cannot wait for the appropriate sessionrestored event, we should add a sleep. Maybe the best location for that is directly in the FirefoxTestCase.restart() method. That way each test will benefit from it.
This failure seems to only happen for the Windows XP node in SCL3. All other Windows machines are working fine. I wonder if it is something related to this machine then. I will see if I can find a local XP machine for tests.
OS: Unspecified → Windows XP
Whiteboard: [test-failure]
Attached file skip patch for Windows XP (obsolete) —
This is the skip patch for Windows XP only. Maybe someone will have the time to investigate. I don't have it at the moment. I will land this change once Travis CI is passing.
The test has been disabled for Windows XP now: (master)

Backports I will do soon once all other failures are fixed, which also need a backport.
The patch as landed doesn't work because the machines report:

version: 5.1.2600
os: win
service_pack: Service Pack 3
bits: 32
processor: x86

That means os_version == "5.1" does not match. I feel this is a bug in handling OS versions in the manifest parser. It should not take the build number into account. I will check this.
The problem is that we haven't had a new release of mozinfo since Oct 2013! Features like os_version are not available via the packages on PyPI. I'm going to release version 0.8 via bug 1169168 to unblock us.
Depends on: 1169168
I also pushed a version bump for mozinfo to 0.8:

If that works I will have to backport both patches down to esr38.
Having our tests for Nightly run in production now, I can also see those timeouts on other versions of Windows like Vista, 7, and 8. So I think we should skip it for all Windows, and check soon what's going on.
OS: Windows XP → Windows
No longer blocks: 1165215
While working on bug 1128968 I noticed that we missed to retrieve the elements for the identity popup again after the restart. That's why it was failing:

I'm not sure why it's working on some platforms but not all. I would wait with further investigation until bug 1177417 has been solved.
Depends on: 1177417
No longer depends on: 1177417
Product: Mozilla QA → Testing
I did a couple of local runs for this test on Windows and I was not able to see this failure anymore. That means I would like to try to unskip this test. A simple backout will not work due to the remote->functional migration. So I will come up with a new patch.
Attachment #8705079 - Flags: review?(mjzffr)
Attachment #8705079 - Flags: review?(mjzffr) → review+
Merged PR to mozilla-central:

If we don't have any failures I would like to also get it enabled for mozilla-aurora which will be the next ESR release.
So all Windows platforms seem to work except for Win XP when we see this failure again now:

I will leave this running for the weekend and check if we constantly fail.
This failure always reproduces on WinXP via the nodes mm-win-xp-32-3 and mm-win-xp-32-4. The permanent issue with OS X 10.10 is bug 1198611. It was working fine since lately. I will check both bugs in more detail tomorrow.
The failure on WinXP was not visible the last two days. Maybe something has been changed here. So it's not permanent. I would leave this test enabled on mozilla-central for now and also enable it in aurora. That way we might get a better idea what's going on if it happens again.

Backported backout patch to mozilla-aurora:
OS: Windows → Windows XP
Summary: Failure in because test doesn't wait after restart for session restored → [WinXP] TEST-UNEXPECTED-ERROR | TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.1 seconds
Whiteboard: [test-failure]
Nowadays this is a permanent failure on both mozilla-central and mozilla-aurora. So we should get this turned off again for WinXP only. I don't think it makes that much sense right now for investigation.
This is similar to attachment 8611467 [details] [review] but with an updated path.
Attachment #8611467 - Attachment is obsolete: true
Attachment #8711633 - Flags: review?(mjzffr)
Adapted patch for mozilla-central on
Attachment #8711635 - Flags: review?(mjzffr)
Attachment #8711633 - Flags: review?(mjzffr) → review+
Attachment #8711635 - Flags: review?(mjzffr) → review+
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: mozilla46 → mozilla47
Attachment 8711635 [details] [diff] is a testonly change and does not affect the product. It constantly fails for us on WinXP, so we would like to have it skipped down to the upcoming 45.0 ESR.
Whiteboard: [checkin-needed-aurora][checkin-needed-beta]
Actually I will mark this bug as wontfix given that it only affects a platform which we can hopefully get rid of in the next months. That's why I don't want to put in any other work.
Resolution: FIXED → WONTFIX
You need to log in before you can comment on or make changes to this bug.