Closed Bug 1334035 Opened 3 years ago Closed 6 months ago

Intermittent TestExecuteContent.test_window_set_timeout_is_not_cancelled | AssertionError: setTimeout already fired


(Testing :: Marionette, defect, P5)

Version 3


(firefox53 disabled, firefox54 disabled, firefox55 disabled)

Tracking Status
firefox53 --- disabled
firefox54 --- disabled
firefox55 --- disabled


(Reporter: intermittent-bug-filer, Unassigned)



(Keywords: intermittent-failure, mobile, Whiteboard: [stockwell disabled])

This is almost Android-only, starting with the bump to tier 2 for Android Marionette tests.

:ato - Could you have a look?
Blocks: 1332205
Flags: needinfo?(ato)
Some more details:

[task 2017-02-16T03:10:27.673685Z] 03:10:27  WARNING -  TEST-UNEXPECTED-FAIL | TestExecuteContent.test_window_set_timeout_is_not_cancelled | AssertionError: setTimeout already fired
[task 2017-02-16T03:10:27.674102Z] 03:10:27     INFO -  Traceback (most recent call last):
[task 2017-02-16T03:10:27.674520Z] 03:10:27     INFO -    File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/marionette_harness/marionette_test/", line 166, in run
[task 2017-02-16T03:10:27.674758Z] 03:10:27     INFO -      testMethod()
[task 2017-02-16T03:10:27.675721Z] 03:10:27     INFO -    File "/home/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/", line 258, in test_window_set_timeout_is_not_cancelled
[task 2017-02-16T03:10:27.675778Z] 03:10:27     INFO -      "setTimeout already fired")

Related test code:

It's definitely a race as described in lines 254 and 255. 

Actually I do not understand what we are testing here. Also which call to execute_script canceled it? Given that Andreas wrote this test, he might be the best person to answer that.
I suspect this is similar to what we saw over in  Please see my comment over there for a general explanation of why we might be seeing timing related issues in slow environments in the comment

Does this occur in Android opt as well as Android debug?  If so, I’m inclined to say we should bump the timeouts up to see if that rectifies the problem.  If it only happens on debug, I’m somewhat inclined to disabling the test because we don’t want to negatively impact opt.

(Tangential: WPT follows a slightly different strategy, whereby all timeout durations are multiplied by 4.  After much experimentation it was found that debug builds on average are between 3-4 times slower than optimised builds.  However, it will be difficult to employ such a fix to Marionette tests because individual tests hard code a lot of numbers.)
Flags: needinfo?(ato)
(In reply to Andreas Tolfsen from comment #5)
> Does this occur in Android opt as well as Android debug?

Only Debug, but that's (maybe) because the Android Marionette test jobs are only run against Debug builds.

Interestingly, I see none of these failures on mozilla-central or integration branches over the last few days.
In that case, I would argue this is also a test we should consider disabling.

Because of the nature of Marionette, and of functional tests in general, I wonder about the benefits of running the Mn job on Android debug at all.  We run debug jobs on all other platforms which should give us sufficient coverage on the non-optimised front.  Since Android debug is so much slower, even than other debug platforms, I wonder if it is worth the time we are spending on diagnosing and adapting the Mn tests to work on Android debug.
No failures on trunk since Feb 16; I don't know why.

If this comes back, on Android, let's disable immediately, as suggested in comment 8.
Assignee: nobody → gbrown
Closed: 3 years ago
Resolution: --- → WORKSFORME
(In reply to Andreas Tolfsen ‹:ato› from comment #8)
> In that case, I would argue this is also a test we should consider disabling.
Resolution: WORKSFORME → ---
Pushed by
Skip Marionette test_window_set_timeout_is_not_cancelled on Android; r=me,test-only
Given the discussion here, I don't see any need to leave-open, but if anyone disagrees, please mark accordingly, or re-open.
Whiteboard: [stockwell disabled]
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
The test is still disabled.
Assignee: gbrown → nobody
Resolution: FIXED → ---
Priority: -- → P5

Lets try to unskip once Mn jobs run with the faster Android 7.0 x86 emulator.

Depends on: 1500509
OS: Unspecified → Android

We moved away from the ARM platform and this failure no longer occurs on with Android 7.0 x86_64.

Closed: 3 years ago6 months ago
Hardware: Unspecified → ARM
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.