1346343 - Investigate spike in main ping submissions

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P1)

Description

[Mark Reid [:mreid]]

Main ping submission rate increased suddenly at 19:00UTC on March 9th[1].

It seems to have increased by 15-20% right on that hour boundary.


[1] https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.trink.doctypes.message_per_minute.html

Comment 1

[Thomas Huelbert]

Mark to spend a point or two, then pass to :whd.

Comment 2

[Mark Reid [:mreid]]

I spent some time investigating this problem today, and came up with a couple of data points.

1. There was a large increase in submissions from the ESR channel at around the same time, which Peter was kind enough to help me track down to the work in https://bugzilla.mozilla.org/show_bug.cgi?id=1318922 . Unfortunately, if I filter out submissions from ESR, the spike still appears plain as ever, so this is not the explanation for the increase in question.

2. There was also an increase in submissions where the "reason" field is set to "environment-change". These submissions *do* seem to correlate very strongly with the observed spike, and are my leading hypothesis for the explanation. So far I don't have any good explanation of why there was an increase in this type of ping.

Comment 3

[Mark Reid [:mreid]]

Attachment: bug1346343.html

I looked further into the increased environment-change pings, and I believe they can be sufficiently explained by a new version release of the firefox-hotfix addon.

I selected 100 clientids from the environment-change pings during the affected period, and used the HBase client history API to fetch that ping and the next one.

There was no notable change in the following fields: app_version, os_version, os_service_pack_major, os_service_pack_minor

I checked addons next, and of the 86 clientids where I was able to fetch the environment change and the subsequent ping, 26 were triggered by either the appearance of the '[e-mail]' addon or a version change in the same addon. In particular, it appears that the addon version string "20170302.01" features heavily.

This matches up with the info from AMO[1]
Version 20170302.01 Info
Last Updated: March 9, 2017

So as far as I can tell, the spike is real and legitimate data.

Case closed.

I've attached a sanitized version of the rendered analysis notebook for reference.

[1] https://addons.mozilla.org/en-US/firefox/addon/firefox-hotfix/versions/?page=1#version-20170302.01