1128135 - Nightly from 2015-01-23 caused +20% increase in Telemetry upload failures

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect)

Description

[Vladan Djeric (:vladan)]

For some reason, Nightly has started to get errors while trying to send its Telemetry pings to the Telemetry server, starting with the Nightly from January 23rd.

The server's submission count graph shows it as well:

http://ec2-50-112-66-71.us-west-2.compute.amazonaws.com:4352/#sandboxes/TelemetryChannelMetrics60DaysAggregator/outputs/TelemetryChannelMetrics60DaysAggregator.nightly.cbuf

We also need to report the HTTP error code in TelemetryLog when the submission fails.

Comment 1

[Vladan Djeric (:vladan)]

Telemetry regression alert:
http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/703/alerts/?from=2015-01-23&to=2015-01-23

Comment 2

[Vladan Djeric (:vladan)]

- It does look like it's related to e10s.. E10S_AUTOSTART ratio went down with the Nightly from January 23rd: http://mzl.la/1Kfv96S
- The fix for missing clientIDs (bug 1119281) is the only TelemetryPing.jsm modification close to Jan 23rd

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

Other histograms have changed the same day, see: http://alerts.telemetry.mozilla.org/index.html#/detectors/1/alerts/?from=2015-01-23&to=2015-01-23&metrics-filter=

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Vladan Djeric (:vladan) from comment #2)
> - It does look like it's related to e10s.. E10S_AUTOSTART ratio went down
> with the Nightly from January 23rd: http://mzl.la/1Kfv96S

Hm, did it go down in volume relative to other measurements?

> - The fix for missing clientIDs (bug 1119281) is the only TelemetryPing.jsm
> modification close to Jan 23rd

Hm, i see a potential shutdown issue there - a missing return here:
https://hg.mozilla.org/mozilla-central/rev/df2f2df820a8#l1.60

Sad that this slipped through :(
I'm not sure if that would account for the dip though.

Comment 5

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> > - The fix for missing clientIDs (bug 1119281) is the only TelemetryPing.jsm
> > modification close to Jan 23rd
> 
> Hm, i see a potential shutdown issue there - a missing return here:
> https://hg.mozilla.org/mozilla-central/rev/df2f2df820a8#l1.60

If that is the issue, we should have recovered on Jan 29 or 30, as bug 1123384 didn't port this as i landed it and we have a return for the AsyncShutdown blocker again:
https://hg.mozilla.org/mozilla-central/annotate/5cbae82bdb4b/toolkit/components/telemetry/TelemetrySession.jsm#l957

Comment 6

[Vladan Djeric (:vladan)]

It actually looks like Nightly submission rates *are* recovering starting on Jan 29th:

http://ec2-50-112-66-71.us-west-2.compute.amazonaws.com:4352/#sandboxes/TelemetryChannelMetrics60DaysAggregator/outputs/TelemetryChannelMetrics60DaysAggregator.nightly.cbuf

Comment 7

[Vladan Djeric (:vladan)]

Nightly submission count increased with Jan 29th build: http://mzl.la/16bSPvA
TELEMETRY_SUCCESS rate went back up to 95% with Jan 29th build: http://mzl.la/1vpvOl2

let's keep an eye on the rates for a couple more days and then we can close this bug

Comment 8

[Georg Fritzsche [:gfritzsche]]

Attachment: Aurora fix for not blocking on promise in TelemetryPing shutdown.

Bug 1119281 was uplifted to Aurora, so we still need to fix this there.

Comment 9

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8558445 [details] [diff] [review]
Aurora fix for not blocking on promise in TelemetryPing shutdown.

Approval Request Comment
[Feature/regressing bug #]: Bug 1119281 regression.
[User impact if declined]: Missing shutdown synchronization and Telemetry submission failures.
[Describe test coverage new/current, TreeHerder]: Obvious fix that restores previous behaviour.
[Risks and why]: Low-risk; obvious fix restoring previous behavior.
[String/UUID change made/needed]: None.

Comment 10

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8558445 [details] [diff] [review]
Aurora fix for not blocking on promise in TelemetryPing shutdown.

Simple fix for Telemetry. Aurora+

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f1c0a517ebc7

Given that bug 1119281 already needs rebasing for Fx36 uplift anyway, I assume that this fix will be rolled into that patch.

Comment 12

[Georg Fritzsche [:gfritzsche]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #11)
> Given that bug 1119281 already needs rebasing for Fx36 uplift anyway, I
> assume that this fix will be rolled into that patch.

Yes, thank you.