Intermittent telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds

RESOLVED FIXED in Firefox 65

Status

()

Product:
Toolkit
Component:
Telemetry
Type:
defect
Importance:
P1
normal
Status:
RESOLVED FIXED
Opened:
9 months ago
Updated:
7 months ago

People

(Reporter: intermittent-bug-filer, Assigned: janerik)

Tracking

({intermittent-failure})

Version:
unspecified
Target:
mozilla66
Points:
1
Duplicates:
1505995

Firefox Tracking Flags

(firefox65 fixed, firefox66 fixed)

Details

Attachments

(2 attachments)

Bug 1505991 - Add new preference to disable ping throttling
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-beta+
Details | Review
Bug 1505991 - Set new testing-only preference to disable Telemetry throttling
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-beta+
Details | Review 
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=210697483&repo=autoland

https://queue.taskcluster.net/v1/task/aAGkQNo2Tjuvv-MEvW72Jw/runs/0/artifacts/public/logs/live_backing.log

00:11:22     INFO -  1541722282614	Marionette	TRACE	1 <- [1,32,null,{"value":"iVBORw0KGgoAAAANSUhEUgAABAAAAAMHCAYAAACnvyW1AAAgAElEQVR4nO3dT2yb953ncR8HO4fd02BuW6DXrZFTDrnMXooAnZPdADmk ... IAAAAABBAAAAAAIIAAAAAAAAEEAAAAAAggAAAAAEAAAQAAAAACCAAAAAAQQAAAAACAAAIAAAAABBAAAAAAIIAAAAAAAAH+AyQLt0odbR2mAAAAAElFTkSuQmCC"}]
00:11:22     INFO -  1541722282617	Marionette	TRACE	1 -> [0,33,"Marionette:SetContext",{"value":"chrome"}]
00:11:22     INFO -  1541722282618	Marionette	TRACE	1 <- [1,33,null,{"value":null}]
00:11:22     INFO -  1541722282620	Marionette	TRACE	1 -> [0,34,"Marionette:GetContext",{}]
00:11:22     INFO -  1541722282621	Marionette	TRACE	1 <- [1,34,null,{"value":"chrome"}]
00:11:22     INFO -  1541722282627	Marionette	TRACE	1 -> [0,35,"Marionette:SetContext",{"value":"content"}]
00:11:22     INFO -  1541722282629	Marionette	TRACE	1 <- [1,35,null,{"value":null}]
00:11:22     INFO -  1541722282631	Marionette	TRACE	1 -> [0,36,"WebDriver:GetPageSource",{}]
00:11:22     INFO -  1541722282639	Marionette	TRACE	1 <- [1,36,null,{"value":"<html><head></head><body></body></html>"}]
00:11:22     INFO -  1541722282641	Marionette	TRACE	1 -> [0,37,"Marionette:SetContext",{"value":"chrome"}]
00:11:22     INFO -  1541722282642	Marionette	TRACE	1 <- [1,37,null,{"value":null}]
00:11:22     INFO - TEST-UNEXPECTED-FAIL | telemetry\marionette\tests\client\test_main_tab_scalars.py TestMainTabScalars.test_main_tab_scalars | AssertionError: Error waiting for ping: Timed out after 60.0 seconds
00:11:22     INFO - Traceback (most recent call last):
00:11:22     INFO -   File "Z:\task_1541721296\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 159, in run
00:11:22     INFO -     testMethod()
00:11:22     INFO -   File "Z:\task_1541721296\build\tests\telemetry\marionette\tests\client\test_main_tab_scalars.py", line 40, in test_main_tab_scalars
00:11:22     INFO -     [ping] = self.wait_for_pings(self.restart_browser, check_is_shutdown_ping, 1)
00:11:22     INFO -   File "Z:\task_1541721296\build\venv\lib\site-packages\telemetry_harness\testcase.py", line 117, in wait_for_pings
00:11:22     INFO -     self.fail("Error waiting for ping: {}".format(e.message))
00:11:22     INFO - TEST-INFO took 84333ms
This looks related to the recent switch to Tier2.
Flags: needinfo?(rpierzina)
Assignee: nobody → rpierzina
Flags: needinfo?(rpierzina)
Duplicate of this bug: 1505995
Priority: P5 → P1
> 00:02:55

We throttle all pings around midnight. We should have a way to change this throttle.
Flags: needinfo?(jrediger)
We put the fuzz delay and timeout in Policy for testing:

https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/app/TelemetrySend.jsm#103-110

Would that do the trick?
Flags: needinfo?(chutten)
Can you please add a code snippet demonstrating how to use this Policy to disable the throttling for the client tests?
Flags: needinfo?(alessio.placitelli)
In order to not lose track of a possible solution (which Raphael and I came up with yesterday):

It seems overriding the Policy per module doesn't work just as expected all the time (because I suspect the JS is running too late), so one other way would be to handle this through a pref (which can be externally set and persists across restarts).

The change in this gist[1] worked locally for me.

[1]: https://gist.github.com/badboy/eac0aff809e293dbc959f0cb1b41b993
Assignee: rpierzina → jrediger
Points: --- → 1
Marionette tests have a hard time properly overwriting the module's
Policies that handle this in xpcshell tests currently.

A preference on the other hand can be easily set from the outside, is
persistent across restarts of the test instance and easily checked.
Pushed by jrediger@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8618ab9657cf
Add new preference to disable ping throttling r=Dexter
https://hg.mozilla.org/integration/autoland/rev/a176164ac2e0
Set new testing-only preference to disable Telemetry throttling r=raphael
https://hg.mozilla.org/mozilla-central/rev/8618ab9657cf
https://hg.mozilla.org/mozilla-central/rev/a176164ac2e0
Status: NEW → RESOLVED
Closed: 7 months ago
status-firefox66: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Please request Beta approval on this when you get a chance. Would be nice to quiet this orange there as well.
status-firefox65: --- → affected
Flags: needinfo?(jrediger)
Comment on attachment 9031179 [details]
Bug 1505991 - Add new preference to disable ping throttling

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: Bug 1382854

User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s.

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only.
When not set the code follows the same path it did before.

String changes made/needed:
Flags: needinfo?(jrediger)
Attachment #9031179 - Flags: approval-mozilla-beta?
Comment on attachment 9031180 [details]
Bug 1505991 - Set new testing-only preference to disable Telemetry throttling

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: Bug 1382854

User impact if declined: None. Beta CI tests might be intermittently orange due to spurious test s.

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Only a new flag, default false, used in a single test only.
When not set the code follows the same path it did before.

String changes made/needed:
Attachment #9031180 - Flags: approval-mozilla-beta?
Comment on attachment 9031179 [details]
Bug 1505991 - Add new preference to disable ping throttling

[Triage Comment]
Fixes a frequent orange in CI. Approved for 65.0b5.
Attachment #9031179 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9031180 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.