1385417 - Intermittent toolkit/components/telemetry/tests/unit/test_TelemetryHealthPing.js | test_sendOnTimeout - [test_sendOnTimeout : 20] Should have recorded a health ping. - "ping-on-timeout" == "health"

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=119077157&repo=autoland

https://queue.taskcluster.net/v1/task/WGwpJ34kQBy0Tdgbh_YBNQ/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

9 failures in 1008 pushes (0.009 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 2

Platform breakdown:
* windows7-32: 8
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1385417&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 2

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Comment 3

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8891948 [details] [diff] [review]
fixTimeout.patch

Review of attachment 8891948 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/components/telemetry/tests/unit/test_TelemetryHealthPing.js
@@ +136,3 @@
>    await TelemetryController.submitExternalPing(PING_TYPE, {});
> +
> +  let ac = new TelemetryArchiveTesting.Checker();

"// Wait for the health ping to be triggered."

@@ +140,5 @@
> +  await waitForConditionWithPromise(() => {
> +    ac.promiseFindPing("health", []);
> +  },"Failed to find health ping", 10);
> +
> +  TelemetrySend.testResetTimeOutToDefault();

"// Reset and trigger ping send activity.

Comment 4

[Georg Fritzsche [:gfritzsche]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ed44c58b985214bea66229ea3023399efd94cbb

Comment 5

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Comment 6

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8892015 [details] [diff] [review]
fixTimeout.patch

Review of attachment 8892015 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/components/telemetry/tests/unit/test_TelemetryHealthPing.js
@@ +31,4 @@
>    telemetryHealthPing.Policy.clearSchedulerTickTimeout = clear;
>  }
>  
> +async function waitForConditionWithPromise(promiseFn, timeoutMsg, tryCount) {

Give this a default value, `tryCount = 30`?

Comment 7

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4446ecfee3d7
Fix sendOnTimeout test. r=gfritzsche

Comment 10

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/6e7e4865a477

That caused near-permaorange (well over 90%) on Win8 in "test_sendOnlyTopTenDiscardedPings - [test_sendOnlyTopTenDiscardedPings : 170] Should have recorded a health ping. - "ping-on-timeout" == "health"" like https://treeherder.mozilla.org/logviewer.html#?job_id=119861691&repo=mozilla-inbound while continuing to intermittently fail on Win7 either the same way, or "test_sendOnTimeout - [test_sendOnTimeout : 21] Should have recorded a health ping. - "ping-on-timeout" == "health"" like https://treeherder.mozilla.org/logviewer.html#?job_id=119822323&repo=mozilla-inbound

(I don't actually need any info from the needinfo, it's just an incredibly stupid thing we do because some people ignore all of their bugmail, including comments on bugs where they are the assignee, and we pretend that's somehow okay.)

Comment 11

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Comment 12

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8892836 [details] [diff] [review]
fixTimeout.patch

Review of attachment 8892836 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/components/telemetry/tests/unit/test_TelemetryHealthPing.js
@@ +127,4 @@
>    PingServer.clearRequests();
>    let PING_TYPE = "ping-on-timeout";
>  
> +  fakePingSendTimer(() => {}, () => {});

Add: // Disable send retry to make this test more deterministic.

@@ +139,2 @@
>      PingServer.resetPingHandler();
> +    res.processAsync();

Add: // We don't finish the response yet to make sure to trigger a timeout.

@@ +156,5 @@
> +  PingServer.resetPingHandler();
> +  TelemetrySend.notifyCanUpload();
> +
> +  let pings = await PingServer.promiseNextPings(2);
> +  let healthPing = pings[0].type === "health" ? pings[0] : pings[1];

... = pings.find(p => p.type == ...);

Then assert that we found a health ping.

Comment 13

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Comment 14

[Georg Fritzsche [:gfritzsche]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e377ff320da2b671726f00657e750124256900f6

Comment 15

[Intermittent Failures Robot]

24 failures in 211 pushes (0.114 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 9
* mozilla-central: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows8-64: 9
* windows10-64: 8
* windows7-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1385417&startday=2017-08-03&endday=2017-08-03&tree=all

Comment 16

[Kate Ustiuzhanina]

Attachment: fixTimeout.patch

Latest try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b46b02713ce97011520408ae4dd524511f3619c

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7164463da2f8
Fix sendOnTimeout test. r=gfritzsche

Comment 18

[Phil Ringnalda (:philor)]

You can add the keyword checkin-needed without removing the keyword intermittent-failure, just stick a comma after the existing one and type another.

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/7164463da2f8

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4923833da3f9

Comment 21

[Intermittent Failures Robot]

67 failures in 888 pushes (0.075 failures/push) were associated with this bug in the last 7 days. 

This is the #18 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 36
* mozilla-inbound: 21
* mozilla-central: 6
* try: 3
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 28
* windows8-64: 22
* windows7-32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1385417&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 22

[Intermittent Failures Robot]

1 failures in 788 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1385417&startday=2018-01-08&endday=2018-01-14&tree=all