Closed Bug 1306848 Opened 7 years ago Closed 4 years ago

Intermittent test_timeouts.py TestTimeouts.test_search_timeout_found | AssertionError: NoSuchElementException not raised

Categories

(Testing :: Marionette Client and Harness, defect, P5)

Product:
Testing
Component:
Marionette Client and Harness
Version:
Version 3
Type:
defect

Tracking

(firefox50 unaffected, firefox51 wontfix, firefox52 wontfix, firefox-esr52 unaffected, firefox-esr60 unaffected, firefox53 disabled, firefox54 disabled, firefox55 disabled, firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox50 --- unaffected
firefox51 --- wontfix
firefox52 --- wontfix
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox53 --- disabled
firefox54 --- disabled
firefox55 --- disabled
firefox67 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell disabled])

Attachments

(1 file)

skip test on Android
887 bytes, patch
ato
: review+
Details | Diff | Splinter Review 
01:13:42    ERROR -  TEST-UNEXPECTED-FAIL | test_timeouts.py TestTimeouts.test_search_timeout_found | AssertionError: NoSuchElementException not raised
01:13:42     INFO -  Traceback (most recent call last):
01:13:42     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test/testcases.py", line 162, in run
01:13:42     INFO -      testMethod()
01:13:42     INFO -    File "/builds/slave/test/build/tests/marionette/tests/testing/marionette/harness/marionette/tests/unit/test_timeouts.py", line 54, in test_search_timeout_found
01:13:42     INFO -      self.assertRaises(NoSuchElementException, self.marionette.find_element, By.ID, "newDiv")

Looks like it is a new failure based on the changes on bug 1302707.
Blocks: 1302707
status-firefox52: --- → affected
Flags: needinfo?(ato)
Keywords: regression
Is this an intermittent or does it show up consistently?
Flags: needinfo?(ato)
OF lists 11 failures starting Oct 1st mostly for debug builds on Linux (all non-e10s related):

https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1306848
Hi :ato,
May I know if you have any updates for this new regression?
Flags: needinfo?(ato)
I don’t have an update.  I see it is still happening even after https://bugzilla.mozilla.org/show_bug.cgi?id=1302707 has landed, so it would require more investigation.
Flags: needinfo?(ato)
This is a regression from making a change to tests; as such, not blocking 51 for the fix.
status-firefox51: affected → fix-optional
This appears to have stopped on trunk around late November (but still hits with low frequency on Aurora/Beta). Any idea what might of fixed it, whimboo? Would be great if we could at least get the fix backported to Aurora for the coming ESR release.
status-firefox53: --- → unaffected
Flags: needinfo?(hskupin)
Orange factor doesn't report any failures for mozilla-central and mozilla-aurora since last November. So I don't see where you are getting this information from. Ryan, do you have some failing reports? If not we are all set for the next ESR.
Flags: needinfo?(hskupin) → needinfo?(ryanvm)
Hm, not sure why I wasn't seeing it when expanding the OF view to Jan 1st. Anyway, it's a single failure by last month and I don't think there is a value to investigate that. Btw right now we uplift all of our changes to aurora to hit the next ESR release. So maybe it got already fixed by one of those uplifts since Dec 8th.
Recent failures here are all Android Debug. I think the re-appearance of this failure is a consequence of bug 1332205, which bumped Marionette-on-Android to tier 2.

Maybe skip this test on Android Debug?
Blocks: 1332205
Flags: needinfo?(hskupin)
The lines of code in question are:

> button = self.marionette.find_element(By.ID, "createDivButton")
> button.click()
> self.assertRaises(NoSuchElementException, self.marionette.find_element, By.ID, "newDiv")

Clicking 'createDivButton' will cause a setTimeout function to execute, which adds 'newDiv' to the DOM after 2s. So I assume that running the test via the emulator we are sooooo slow, that this method actually executes before we come to the 'find_element' call.

Andreas, mind telling us what this test should actually test nowadays? As I mentioned earlier it might have been started to fail with the changes on bug 1302707.
Flags: needinfo?(hskupin) → needinfo?(ato)
The test TestTimeouts.test_search_timeout_found is badly written.  Its intention is to test that setting the implicit search timeout to 0 will cause element retrieval of div#newDiv to fail because it is only available in the document two seconds after clicking createDivButton.  However, it does not explicitly set the implicit search timeout and relies on the default value which just happens to be 0 in this case.

If we are seeing NoSuchElementException being raised on Android debug, the only explanation is that div#newDiv appears in the document before we have time to call marionette.find_element.  This could very well be explained by the fact that Android debug with an emulator is slow, but there is really no fix for that in this case.

Marionette provides out-of-process remote control of Gecko, so if the turnaround between button.click and marionette.find_element is greater than two seconds, then there’s really nothing we can do about it.  The only thing that would mitigate this would be to raise the two second setTimeout of createDivButton and the implicit wait durations used in test_timeouts.py in hope that the environment will have time to process everything before the final call to marionette.find_element for div#newDiv.  Unfortunately, this would have the side effect of also making test execution on all other platforms slower.

We could either disable this test on Android debug, and I don’t think that would be so bad, or we could raise the timeouts.  I am happy with either.  gbrown, what do you want to do?
Flags: needinfo?(ato) → needinfo?(gbrown)
(In reply to Andreas Tolfsen from comment #21)
> We could either disable this test on Android debug, and I don’t think that
> would be so bad, or we could raise the timeouts.  I am happy with either. 
> gbrown, what do you want to do?

I don't have a strong opinion, but have a slight preference for skipping this test on Android (no need to distinguish Debug -- Marionette on Android only runs against Debug builds), rather than trying to find the "right" timeout value.
Flags: needinfo?(gbrown)
I assume this test fixed itself as I see no reference to a patch which skipped this test, yet our failure rate went down significantly.
Whiteboard: [stockwell unknown]
Assignee: nobody → gbrown
See Also: → 1317121
Follow-up to comment 21, since this keeps coming back.
Attachment #8855367 - Flags: review?(ato)
Attachment #8855367 - Flags: review?(ato) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/af12e840ff41
Skip TestTimeouts.test_search_timeout_found on Android; r=ato
Assignee: gbrown → nobody
Keywords: leave-open
Whiteboard: [stockwell unknown] → [stockwell disabled]
Priority: -- → P5

This test got re-enabled on Android with bug 1533079. The move to Android x86_64 made it possible.

Assignee: nobody → hskupin
Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox67: --- → fixed
Depends on: 1533079, 1500509
Keywords: leave-open
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing
You need to log in before you can comment on or make changes to this bug.