1315981 - t.m.o submission by build_id dropoff since Nov 2nd 2016

Status: RESOLVED WORKSFORME

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Mike Conley (:mconley) (:⚙️)]

Open up telemetry.mozilla.org, and look at a metric of your choice for Nightly, and make sure to chart by build_id on the X axis.

There's a noticeable dropoff in submission on Nov 2nd, and it hasn't recovered yet. According to chutten's Telemetry-based crash dashboard, the measurement of kuh (1000 usage hours) hasn't changed... so I'm placing my bet on pipline to t.m.o.

Here's an example chart:

https://mzl.la/2eAmVxh

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Here's chutten's Telemetry-based crash dashboard, for reference: https://chutten.github.io/telemetry_crashes/

Comment 2

[Mike Conley (:mconley) (:⚙️)]

Also, the drop off is by about 50%, from the probes I've been looking at.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Actually, one thing to note is that if you plot the graph with the submission date along the X axis, there's no drop-off.

So perhaps this is an update saturation problem.

Comment 4

[Chris H-C :chutten]

Here's what I did: https://gist.github.com/chutten/3ac71bc6fde892eaf489337394dfaf84

There may have been a similar sort of effect last month, but it isn't as obvious as this month's. More analysis likely needed.

Comment 5

[Frank Bertsch [:frank]]

Went ahead and looked at the distribution Submission Date - Build Date. Caution, these queries have NOT been reviewed.

The first [0] just takes all the nightly data and plots the distribution. It didn't seem as skewed left as I would have supposed, so I unweighted the users (i.e. each ping is only "worth" 1/count(pings), rather than all weighted equally - this weight heavy users don't skew results). That is available in [1]. It did move the 50th percentile a bit left, but not much effect other than that.

These results seem a bit surprising to me, it takes about a month before we see 50% of pings for a nightly build.

[0] https://sql.telemetry.mozilla.org/queries/1619/source#2870
[1] https://sql.telemetry.mozilla.org/queries/1622/source#2872

Comment 6

[Chris H-C :chutten]

We get a full third of pings within two days, generally speaking, which is neat.

This is an average over all pings, I presume? I wonder if there are effects that move a given build's curve left or right...

Comment 7

[Thomas Huelbert]

Frank - can this be closed?

Comment 8

[Frank Bertsch [:frank]]

Yes, we'll keep an eye on this, but seems to be okay for now.