Intermittent telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds
RESOLVED
FIXED
in Firefox 65
Status
()
People
(Reporter: intermittent-bug-filer, Assigned: janerik)
Tracking
({intermittent-failure})
Firefox Tracking Flags
(firefox65 fixed, firefox66 fixed)
Details
Attachments
(2 attachments)
|
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
|
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
Filed by: ncsoregi [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=210697483&repo=autoland https://queue.taskcluster.net/v1/task/aAGkQNo2Tjuvv-MEvW72Jw/runs/0/artifacts/public/logs/live_backing.log 00:11:22 INFO - 1541722282614 Marionette TRACE 1 <- [1,32,null,{"value":"iVBORw0KGgoAAAANSUhEUgAABAAAAAMHCAYAAACnvyW1AAAgAElEQVR4nO3dT2yb953ncR8HO4fd02BuW6DXrZFTDrnMXooAnZPdADmk ... IAAAAABBAAAAAAIIAAAAAAAAEEAAAAAAggAAAAAEAAAQAAAAACCAAAAAAQQAAAAACAAAIAAAAABBAAAAAAIIAAAAAAAAH+AyQLt0odbR2mAAAAAElFTkSuQmCC"}] 00:11:22 INFO - 1541722282617 Marionette TRACE 1 -> [0,33,"Marionette:SetContext",{"value":"chrome"}] 00:11:22 INFO - 1541722282618 Marionette TRACE 1 <- [1,33,null,{"value":null}] 00:11:22 INFO - 1541722282620 Marionette TRACE 1 -> [0,34,"Marionette:GetContext",{}] 00:11:22 INFO - 1541722282621 Marionette TRACE 1 <- [1,34,null,{"value":"chrome"}] 00:11:22 INFO - 1541722282627 Marionette TRACE 1 -> [0,35,"Marionette:SetContext",{"value":"content"}] 00:11:22 INFO - 1541722282629 Marionette TRACE 1 <- [1,35,null,{"value":null}] 00:11:22 INFO - 1541722282631 Marionette TRACE 1 -> [0,36,"WebDriver:GetPageSource",{}] 00:11:22 INFO - 1541722282639 Marionette TRACE 1 <- [1,36,null,{"value":"<html><head></head><body></body></html>"}] 00:11:22 INFO - 1541722282641 Marionette TRACE 1 -> [0,37,"Marionette:SetContext",{"value":"chrome"}] 00:11:22 INFO - 1541722282642 Marionette TRACE 1 <- [1,37,null,{"value":null}] 00:11:22 INFO - TEST-UNEXPECTED-FAIL | telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds 00:11:22 INFO - Traceback (most recent call last): 00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 159, in run 00:11:22 INFO - testMethod() 00:11:22 INFO - File "Z:\task_1541721296\build\tests\telemetry\marionette\tests\client\test_main_tab_scalars.py", line 40, in test_main_tab_scalars 00:11:22 INFO - [ping] = self.wait_for_pings(self.restart_browser, check_is_shutdown_ping, 1) 00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\telemetry_harness\testcase.py", line 117, in wait_for_pings 00:11:22 INFO - self.fail("Error waiting for ping: {}".format(e.message)) 00:11:22 INFO - TEST-INFO took 84333ms
|
||
Comment 1•7 months ago
|
||
This looks related to the recent switch to Tier2.
Flags: needinfo?(rpierzina)
|
||
Updated•7 months ago
|
Assignee: nobody → rpierzina
Flags: needinfo?(rpierzina)
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
|
|
||
Updated•6 months ago
|
Priority: P5 → P1
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
|
|
||
Comment 8•6 months ago
|
||
Looking at the CI logs it appears TelemetrySend is getting throttled and delays sending out the "shutdown" main ping by several minutes, resulting in the test to time out after 60 seconds. https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=6cbbac93c27318820e9a626a25aced587ef3a010&selectedJob=215221226 https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215221226&repo=autoland&lineNumber=1821 https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=6cbbac93c27318820e9a626a25aced587ef3a010&selectedJob=215222468 https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215222468&repo=autoland&lineNumber=2220 https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=847bc008e00b9348a07b87b4c8063782c1d7ab11&selectedJob=214993460 https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=214993460&repo=autoland&lineNumber=2686 Any idea why this is happening and how to get this to work for the client test?
Flags: needinfo?(jrediger)
Flags: needinfo?(chutten)
Flags: needinfo?(alessio.placitelli)
|
Assignee | |
Comment 9•6 months ago
|
||
> 00:02:55
We throttle all pings around midnight. We should have a way to change this throttle.Flags: needinfo?(jrediger)
|
||
Comment 10•6 months ago
|
||
We put the fuzz delay and timeout in Policy for testing: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/app/TelemetrySend.jsm#103-110 Would that do the trick?
Flags: needinfo?(chutten)
|
|
||
Comment 11•6 months ago
|
||
Can you please add a code snippet demonstrating how to use this Policy to disable the throttling for the client tests?
|
|
||
Comment 12•6 months ago
|
||
There are some utility functions in our xpcshell head.js here: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#259-269 They set the default for xpcshell tests to 0ms of fuzzing and immediate timer firing: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#392-395
|
|
||
Updated•6 months ago
|
Flags: needinfo?(alessio.placitelli)
| Comment hidden (Intermittent Failures Robot) |
|
|
Assignee | |
Comment 14•5 months ago
|
||
In order to not lose track of a possible solution (which Raphael and I came up with yesterday): It seems overriding the Policy per module doesn't work just as expected all the time (because I suspect the JS is running too late), so one other way would be to handle this through a pref (which can be externally set and persists across restarts). The change in this gist[1] worked locally for me. [1]: https://gist.github.com/badboy/eac0aff809e293dbc959f0cb1b41b993
|
|
Assignee | |
Updated•5 months ago
|
Assignee: rpierzina → jrediger
Points: --- → 1
|
|
Assignee | |
Comment 15•5 months ago
|
||
Marionette tests have a hard time properly overwriting the module's Policies that handle this in xpcshell tests currently. A preference on the other hand can be easily set from the outside, is persistent across restarts of the test instance and easily checked.
|
|
Assignee | |
Comment 16•5 months ago
|
||
Depends on D14458
|
||
Comment 17•5 months ago
|
||
Pushed by jrediger@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8618ab9657cf Add new preference to disable ping throttling r=Dexter https://hg.mozilla.org/integration/autoland/rev/a176164ac2e0 Set new testing-only preference to disable Telemetry throttling r=raphael
|
||
Comment 18•5 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/8618ab9657cf https://hg.mozilla.org/mozilla-central/rev/a176164ac2e0
Status: NEW → RESOLVED
Last Resolved: 5 months ago
status-firefox66:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
|
||
Comment 19•5 months ago
|
||
Please request Beta approval on this when you get a chance. Would be nice to quiet this orange there as well.
status-firefox65:
--- → affected
Flags: needinfo?(jrediger)
|
|
Assignee | |
Comment 20•5 months ago
|
||
Comment on attachment 9031179 [details] Bug 1505991 - Add new preference to disable ping throttling [Beta/Release Uplift Approval Request] Feature/Bug causing the regression: Bug 1382854 User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s. Is this code covered by automated tests?: Yes Has the fix been verified in Nightly?: Yes Needs manual test from QE?: No If yes, steps to reproduce: List of other uplifts needed: None Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only. When not set the code follows the same path it did before. String changes made/needed:
Flags: needinfo?(jrediger)
Attachment #9031179 -
Flags: approval-mozilla-beta?
|
|
Assignee | |
Comment 21•5 months ago
|
||
Comment on attachment 9031180 [details] Bug 1505991 - Set new testing-only preference to disable Telemetry throttling [Beta/Release Uplift Approval Request] Feature/Bug causing the regression: Bug 1382854 User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s. Is this code covered by automated tests?: Yes Has the fix been verified in Nightly?: Yes Needs manual test from QE?: No If yes, steps to reproduce: List of other uplifts needed: None Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only. When not set the code follows the same path it did before. String changes made/needed:
Attachment #9031180 -
Flags: approval-mozilla-beta?
|
|
||
Comment 22•5 months ago
|
||
Comment on attachment 9031179 [details] Bug 1505991 - Add new preference to disable ping throttling [Triage Comment] Fixes a frequent orange in CI. Approved for 65.0b5.
Attachment #9031179 -
Flags: approval-mozilla-beta? → approval-mozilla-beta+
|
|
||
Updated•5 months ago
|
Attachment #9031180 -
Flags: approval-mozilla-beta? → approval-mozilla-beta+
|
||
Comment 23•5 months ago
|
||
| bugherderuplift | ||
https://hg.mozilla.org/releases/mozilla-beta/rev/34ce66233646 https://hg.mozilla.org/releases/mozilla-beta/rev/e72979e80a1c
| Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•