Closed Bug 1306848 Opened 4 years ago Closed 1 year ago
_timeouts .py Test Timeouts .test _search _timeout _found | Assertion Error: No Such Element Exception not raised
Filed by: philringnalda [at] gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=5127567&repo=mozilla-central https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux-debug/1475303095/mozilla-central_ubuntu32_vm-debug_test-marionette-bm04-tests1-linux32-build21.txt.gz
01:13:42 ERROR - TEST-UNEXPECTED-FAIL | test_timeouts.py TestTimeouts.test_search_timeout_found | AssertionError: NoSuchElementException not raised 01:13:42 INFO - Traceback (most recent call last): 01:13:42 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test/testcases.py", line 162, in run 01:13:42 INFO - testMethod() 01:13:42 INFO - File "/builds/slave/test/build/tests/marionette/tests/testing/marionette/harness/marionette/tests/unit/test_timeouts.py", line 54, in test_search_timeout_found 01:13:42 INFO - self.assertRaises(NoSuchElementException, self.marionette.find_element, By.ID, "newDiv") Looks like it is a new failure based on the changes on bug 1302707.
Is this an intermittent or does it show up consistently?
OF lists 11 failures starting Oct 1st mostly for debug builds on Linux (all non-e10s related): https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1306848
Hi :ato, May I know if you have any updates for this new regression?
I don’t have an update. I see it is still happening even after https://bugzilla.mozilla.org/show_bug.cgi?id=1302707 has landed, so it would require more investigation.
This is a regression from making a change to tests; as such, not blocking 51 for the fix.
This appears to have stopped on trunk around late November (but still hits with low frequency on Aurora/Beta). Any idea what might of fixed it, whimboo? Would be great if we could at least get the fix backported to Aurora for the coming ESR release.
Orange factor doesn't report any failures for mozilla-central and mozilla-aurora since last November. So I don't see where you are getting this information from. Ryan, do you have some failing reports? If not we are all set for the next ESR.
Flags: needinfo?(hskupin) → needinfo?(ryanvm)
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1306848&endday=2017-01-03&startday=2016-11-01&tree=mozilla-aurora shows a hit in December too?
Hm, not sure why I wasn't seeing it when expanding the OF view to Jan 1st. Anyway, it's a single failure by last month and I don't think there is a value to investigate that. Btw right now we uplift all of our changes to aurora to hit the next ESR release. So maybe it got already fixed by one of those uplifts since Dec 8th.
Recent failures here are all Android Debug. I think the re-appearance of this failure is a consequence of bug 1332205, which bumped Marionette-on-Android to tier 2. Maybe skip this test on Android Debug?
The lines of code in question are: > button = self.marionette.find_element(By.ID, "createDivButton") > button.click() > self.assertRaises(NoSuchElementException, self.marionette.find_element, By.ID, "newDiv") Clicking 'createDivButton' will cause a setTimeout function to execute, which adds 'newDiv' to the DOM after 2s. So I assume that running the test via the emulator we are sooooo slow, that this method actually executes before we come to the 'find_element' call. Andreas, mind telling us what this test should actually test nowadays? As I mentioned earlier it might have been started to fail with the changes on bug 1302707.
Flags: needinfo?(hskupin) → needinfo?(ato)
The test TestTimeouts.test_search_timeout_found is badly written. Its intention is to test that setting the implicit search timeout to 0 will cause element retrieval of div#newDiv to fail because it is only available in the document two seconds after clicking createDivButton. However, it does not explicitly set the implicit search timeout and relies on the default value which just happens to be 0 in this case. If we are seeing NoSuchElementException being raised on Android debug, the only explanation is that div#newDiv appears in the document before we have time to call marionette.find_element. This could very well be explained by the fact that Android debug with an emulator is slow, but there is really no fix for that in this case. Marionette provides out-of-process remote control of Gecko, so if the turnaround between button.click and marionette.find_element is greater than two seconds, then there’s really nothing we can do about it. The only thing that would mitigate this would be to raise the two second setTimeout of createDivButton and the implicit wait durations used in test_timeouts.py in hope that the environment will have time to process everything before the final call to marionette.find_element for div#newDiv. Unfortunately, this would have the side effect of also making test execution on all other platforms slower. We could either disable this test on Android debug, and I don’t think that would be so bad, or we could raise the timeouts. I am happy with either. gbrown, what do you want to do?
Flags: needinfo?(ato) → needinfo?(gbrown)
(In reply to Andreas Tolfsen from comment #21) > We could either disable this test on Android debug, and I don’t think that > would be so bad, or we could raise the timeouts. I am happy with either. > gbrown, what do you want to do? I don't have a strong opinion, but have a slight preference for skipping this test on Android (no need to distinguish Debug -- Marionette on Android only runs against Debug builds), rather than trying to find the "right" timeout value.
I assume this test fixed itself as I see no reference to a patch which skipped this test, yet our failure rate went down significantly.
Whiteboard: [stockwell unknown]
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/af12e840ff41 Skip TestTimeouts.test_search_timeout_found on Android; r=ato
Assignee: gbrown → nobody
Whiteboard: [stockwell unknown] → [stockwell disabled]
You need to log in before you can comment on or make changes to this bug.