Closed Bug 1505991 Opened 7 years ago Closed 7 years ago

Intermittent telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds

Categories

(Toolkit :: Telemetry, defect, P1)

Product:
Toolkit
Component:
Telemetry
Type:
defect
Points:
1

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla66
Tracking Flags:
Tracking Status
firefox65 --- fixed
firefox66 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: janerik)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

Bug 1505991 - Add new preference to disable ping throttling
47 bytes, text/x-phabricator-request
ryanvm
: approval-mozilla-beta+
Details | Review
Bug 1505991 - Set new testing-only preference to disable Telemetry throttling
47 bytes, text/x-phabricator-request
ryanvm
: approval-mozilla-beta+
Details | Review
Filed by: ncsoregi [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=210697483&repo=autoland https://queue.taskcluster.net/v1/task/aAGkQNo2Tjuvv-MEvW72Jw/runs/0/artifacts/public/logs/live_backing.log 00:11:22 INFO - 1541722282614 Marionette TRACE 1 <- [1,32,null,{"value":"iVBORw0KGgoAAAANSUhEUgAABAAAAAMHCAYAAACnvyW1AAAgAElEQVR4nO3dT2yb953ncR8HO4fd02BuW6DXrZFTDrnMXooAnZPdADmk ... IAAAAABBAAAAAAIIAAAAAAAAEEAAAAAAggAAAAAEAAAQAAAAACCAAAAAAQQAAAAACAAAIAAAAABBAAAAAAIIAAAAAAAAH+AyQLt0odbR2mAAAAAElFTkSuQmCC"}] 00:11:22 INFO - 1541722282617 Marionette TRACE 1 -> [0,33,"Marionette:SetContext",{"value":"chrome"}] 00:11:22 INFO - 1541722282618 Marionette TRACE 1 <- [1,33,null,{"value":null}] 00:11:22 INFO - 1541722282620 Marionette TRACE 1 -> [0,34,"Marionette:GetContext",{}] 00:11:22 INFO - 1541722282621 Marionette TRACE 1 <- [1,34,null,{"value":"chrome"}] 00:11:22 INFO - 1541722282627 Marionette TRACE 1 -> [0,35,"Marionette:SetContext",{"value":"content"}] 00:11:22 INFO - 1541722282629 Marionette TRACE 1 <- [1,35,null,{"value":null}] 00:11:22 INFO - 1541722282631 Marionette TRACE 1 -> [0,36,"WebDriver:GetPageSource",{}] 00:11:22 INFO - 1541722282639 Marionette TRACE 1 <- [1,36,null,{"value":"<html><head></head><body></body></html>"}] 00:11:22 INFO - 1541722282641 Marionette TRACE 1 -> [0,37,"Marionette:SetContext",{"value":"chrome"}] 00:11:22 INFO - 1541722282642 Marionette TRACE 1 <- [1,37,null,{"value":null}] 00:11:22 INFO - TEST-UNEXPECTED-FAIL | telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds 00:11:22 INFO - Traceback (most recent call last): 00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 159, in run 00:11:22 INFO - testMethod() 00:11:22 INFO - File "Z:\task_1541721296\build\tests\telemetry\marionette\tests\client\test_main_tab_scalars.py", line 40, in test_main_tab_scalars 00:11:22 INFO - [ping] = self.wait_for_pings(self.restart_browser, check_is_shutdown_ping, 1) 00:11:22 INFO - File "Z:\task_1541721296\build\venv\lib\site-packages\telemetry_harness\testcase.py", line 117, in wait_for_pings 00:11:22 INFO - self.fail("Error waiting for ping: {}".format(e.message)) 00:11:22 INFO - TEST-INFO took 84333ms
This looks related to the recent switch to Tier2.
Flags: needinfo?(rpierzina)
Assignee: nobody → rpierzina
Flags: needinfo?(rpierzina)
Priority: P5 → P1
> 00:02:55 We throttle all pings around midnight. We should have a way to change this throttle.
Flags: needinfo?(jrediger)
We put the fuzz delay and timeout in Policy for testing: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/app/TelemetrySend.jsm#103-110 Would that do the trick?
Flags: needinfo?(chutten)
Can you please add a code snippet demonstrating how to use this Policy to disable the throttling for the client tests?
There are some utility functions in our xpcshell head.js here: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#259-269 They set the default for xpcshell tests to 0ms of fuzzing and immediate timer firing: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/head.js#392-395
Flags: needinfo?(alessio.placitelli)
In order to not lose track of a possible solution (which Raphael and I came up with yesterday): It seems overriding the Policy per module doesn't work just as expected all the time (because I suspect the JS is running too late), so one other way would be to handle this through a pref (which can be externally set and persists across restarts). The change in this gist[1] worked locally for me. [1]: https://gist.github.com/badboy/eac0aff809e293dbc959f0cb1b41b993
Assignee: rpierzina → jrediger
Points: --- → 1
Marionette tests have a hard time properly overwriting the module's Policies that handle this in xpcshell tests currently. A preference on the other hand can be easily set from the outside, is persistent across restarts of the test instance and easily checked.
Pushed by jrediger@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8618ab9657cf Add new preference to disable ping throttling r=Dexter https://hg.mozilla.org/integration/autoland/rev/a176164ac2e0 Set new testing-only preference to disable Telemetry throttling r=raphael
https://hg.mozilla.org/mozilla-central/rev/8618ab9657cf https://hg.mozilla.org/mozilla-central/rev/a176164ac2e0
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox66: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Please request Beta approval on this when you get a chance. Would be nice to quiet this orange there as well.
status-firefox65: --- → affected
Flags: needinfo?(jrediger)
Comment on attachment 9031179 [details] Bug 1505991 - Add new preference to disable ping throttling [Beta/Release Uplift Approval Request] Feature/Bug causing the regression: Bug 1382854 User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s. Is this code covered by automated tests?: Yes Has the fix been verified in Nightly?: Yes Needs manual test from QE?: No If yes, steps to reproduce: List of other uplifts needed: None Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only. When not set the code follows the same path it did before. String changes made/needed:
Flags: needinfo?(jrediger)
Attachment #9031179 - Flags: approval-mozilla-beta?
Comment on attachment 9031180 [details] Bug 1505991 - Set new testing-only preference to disable Telemetry throttling [Beta/Release Uplift Approval Request] Feature/Bug causing the regression: Bug 1382854 User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s. Is this code covered by automated tests?: Yes Has the fix been verified in Nightly?: Yes Needs manual test from QE?: No If yes, steps to reproduce: List of other uplifts needed: None Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only. When not set the code follows the same path it did before. String changes made/needed:
Attachment #9031180 - Flags: approval-mozilla-beta?
Comment on attachment 9031179 [details] Bug 1505991 - Add new preference to disable ping throttling [Triage Comment] Fixes a frequent orange in CI. Approved for 65.0b5.
Attachment #9031179 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9031180 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: