Closed
Bug 1505991
Opened 7 years ago
Closed 7 years ago
Intermittent telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds
Categories
(Toolkit :: Telemetry, defect, P1)
Toolkit
Telemetry
Tracking
()
RESOLVED
FIXED
mozilla66
People
(Reporter: intermittent-bug-filer, Assigned: janerik)
References
Details
(Keywords: intermittent-failure)
Attachments
(2 files)
47 bytes,
text/x-phabricator-request
|
ryanvm
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
ryanvm
:
approval-mozilla-beta+
|
Details | Review |
Filed by: ncsoregi [at] mozilla.com
https://treeherder.mozilla.org/logviewer.html#?job_id=210697483&repo=autoland
https://queue.taskcluster.net/v1/task/aAGkQNo2Tjuvv-MEvW72Jw/runs/0/artifacts/public/logs/live_backing.log
00:11:22 INFO - 1541722282614 Marionette TRACE 1 <- [1,32,null,{"value":"iVBORw0KGgoAAAANSUhEUgAABAAAAAMHCAYAAACnvyW1AAAgAElEQVR4nO3dT2yb953ncR8HO4fd02BuW6DXrZFTDrnMXooAnZPdADmk ... IAAAAABBAAAAAAIIAAAAAAAAEEAAAAAAggAAAAAEAAAQAAAAACCAAAAAAQQAAAAACAAAIAAAAABBAAAAAAIIAAAAAAAAH+AyQLt0odbR2mAAAAAElFTkSuQmCC"}]
00:11:22 INFO - 1541722282617 Marionette TRACE 1 -> [0,33,"Marionette:SetContext",{"value":"chrome"}]
00:11:22 INFO - 1541722282618 Marionette TRACE 1 <- [1,33,null,{"value":null}]
00:11:22 INFO - 1541722282620 Marionette TRACE 1 -> [0,34,"Marionette:GetContext",{}]
00:11:22 INFO - 1541722282621 Marionette TRACE 1 <- [1,34,null,{"value":"chrome"}]
00:11:22 INFO - 1541722282627 Marionette TRACE 1 -> [0,35,"Marionette:SetContext",{"value":"content"}]
00:11:22 INFO - 1541722282629 Marionette TRACE 1 <- [1,35,null,{"value":null}]
00:11:22 INFO - 1541722282631 Marionette TRACE 1 -> [0,36,"WebDriver:GetPageSource",{}]
00:11:22 INFO - 1541722282639 Marionette TRACE 1 <- [1,36,null,{"value":"<html><head></head><body></body></html>"}]
00:11:22 INFO - 1541722282641 Marionette TRACE 1 -> [0,37,"Marionette:SetContext",{"value":"chrome"}]
00:11:22 INFO - 1541722282642 Marionette TRACE 1 <- [1,37,null,{"value":null}]
00:11:22 INFO - TEST-UNEXPECTED-FAIL | telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds
00:11:22 INFO - Traceback (most recent call last):
00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 159, in run
00:11:22 INFO - testMethod()
00:11:22 INFO - File "Z:\task_1541721296\build\tests\telemetry\marionette\tests\client\test_main_tab_scalars.py", line 40, in test_main_tab_scalars
00:11:22 INFO - [ping] = self.wait_for_pings(self.restart_browser, check_is_shutdown_ping, 1)
00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\telemetry_harness\testcase.py", line 117, in wait_for_pings
00:11:22 INFO - self.fail("Error waiting for ping: {}".format(e.message))
00:11:22 INFO - TEST-INFO took 84333ms
Comment 1•7 years ago
|
||
This looks related to the recent switch to Tier2.
Flags: needinfo?(rpierzina)
Updated•7 years ago
|
Assignee: nobody → rpierzina
Flags: needinfo?(rpierzina)
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•7 years ago
|
Priority: P5 → P1
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 8•7 years ago
|
||
Looking at the CI logs it appears TelemetrySend is getting throttled and delays sending out the "shutdown" main ping by several minutes, resulting in the test to time out after 60 seconds.
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=6cbbac93c27318820e9a626a25aced587ef3a010&selectedJob=215221226
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215221226&repo=autoland&lineNumber=1821
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=6cbbac93c27318820e9a626a25aced587ef3a010&selectedJob=215222468
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215222468&repo=autoland&lineNumber=2220
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=847bc008e00b9348a07b87b4c8063782c1d7ab11&selectedJob=214993460
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=214993460&repo=autoland&lineNumber=2686
Any idea why this is happening and how to get this to work for the client test?
Flags: needinfo?(jrediger)
Flags: needinfo?(chutten)
Flags: needinfo?(alessio.placitelli)
Assignee | ||
Comment 9•7 years ago
|
||
> 00:02:55
We throttle all pings around midnight. We should have a way to change this throttle.
Flags: needinfo?(jrediger)
Comment 10•7 years ago
|
||
We put the fuzz delay and timeout in Policy for testing:
https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/app/TelemetrySend.jsm#103-110
Would that do the trick?
Flags: needinfo?(chutten)
Comment 11•7 years ago
|
||
Can you please add a code snippet demonstrating how to use this Policy to disable the throttling for the client tests?
Comment 12•7 years ago
|
||
There are some utility functions in our xpcshell head.js here: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#259-269
They set the default for xpcshell tests to 0ms of fuzzing and immediate timer firing: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#392-395
Updated•7 years ago
|
Flags: needinfo?(alessio.placitelli)
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 14•7 years ago
|
||
In order to not lose track of a possible solution (which Raphael and I came up with yesterday):
It seems overriding the Policy per module doesn't work just as expected all the time (because I suspect the JS is running too late), so one other way would be to handle this through a pref (which can be externally set and persists across restarts).
The change in this gist[1] worked locally for me.
[1]: https://gist.github.com/badboy/eac0aff809e293dbc959f0cb1b41b993
Assignee | ||
Updated•7 years ago
|
Assignee: rpierzina → jrediger
Points: --- → 1
Assignee | ||
Comment 15•7 years ago
|
||
Marionette tests have a hard time properly overwriting the module's
Policies that handle this in xpcshell tests currently.
A preference on the other hand can be easily set from the outside, is
persistent across restarts of the test instance and easily checked.
Assignee | ||
Comment 16•7 years ago
|
||
Depends on D14458
Comment 17•7 years ago
|
||
Pushed by jrediger@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8618ab9657cf
Add new preference to disable ping throttling r=Dexter
https://hg.mozilla.org/integration/autoland/rev/a176164ac2e0
Set new testing-only preference to disable Telemetry throttling r=raphael
Comment 18•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/8618ab9657cf
https://hg.mozilla.org/mozilla-central/rev/a176164ac2e0
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox66:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Comment 19•7 years ago
|
||
Please request Beta approval on this when you get a chance. Would be nice to quiet this orange there as well.
status-firefox65:
--- → affected
Flags: needinfo?(jrediger)
Assignee | ||
Comment 20•7 years ago
|
||
Comment on attachment 9031179 [details]
Bug 1505991 - Add new preference to disable ping throttling
[Beta/Release Uplift Approval Request]
Feature/Bug causing the regression: Bug 1382854
User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s.
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: Yes
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed: None
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only.
When not set the code follows the same path it did before.
String changes made/needed:
Flags: needinfo?(jrediger)
Attachment #9031179 -
Flags: approval-mozilla-beta?
Assignee | ||
Comment 21•7 years ago
|
||
Comment on attachment 9031180 [details]
Bug 1505991 - Set new testing-only preference to disable Telemetry throttling
[Beta/Release Uplift Approval Request]
Feature/Bug causing the regression: Bug 1382854
User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s.
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: Yes
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed: None
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only.
When not set the code follows the same path it did before.
String changes made/needed:
Attachment #9031180 -
Flags: approval-mozilla-beta?
Comment 22•7 years ago
|
||
Comment on attachment 9031179 [details]
Bug 1505991 - Add new preference to disable ping throttling
[Triage Comment]
Fixes a frequent orange in CI. Approved for 65.0b5.
Attachment #9031179 -
Flags: approval-mozilla-beta? → approval-mozilla-beta+
Updated•7 years ago
|
Attachment #9031180 -
Flags: approval-mozilla-beta? → approval-mozilla-beta+
Comment 23•7 years ago
|
||
bugherder uplift |
Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•