1246722 - Funnelcake identifier and/or data missing in UT

Status: VERIFIED WORKSFORME

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Funnelcake builds use a special identifier for appUpdateChannel, for example, instead of "release", we would see a value such as "release-cck-mozilla51" for the channel.

While working on bug 1226379, it surfaced that we're not seeing recent Funnelcake builds in Unified Telemetry data.

Let's track this down and make sure we're sending and receiving data correctly for these builds.

Comment 1

[Katie Parlante]

Benjamin mentioned that there may have been a recent fix where the channel is set to "release" for these builds.

He suggested creating a build and looking at the output.

Comment 2

[Mark Reid [:mreid]]

Benjamin, do you know if there is a different way we should be identifying Funnelcake builds in Unified Telemetry?

Comment 3

[Benjamin Smedberg]

You should verify with actual testing, but environment.partner.distributionId is supposed to be the canonical way to identify these.

Comment 4

[Georg Fritzsche [:gfritzsche]]

Bug 1227428 is related here, which fixed missing partner information.
From Fx43 on we should be able to identify those builds by environment.partner.distributionId.

Comment 5

[Chris More [:cmore]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #3)
> You should verify with actual testing, but
> environment.partner.distributionId is supposed to be the canonical way to
> identify these.

Yeah, that would be the ultimate way to test as we could just distribute another vanilla funnelcake and ensure we can measure it in UT. Let's wait until we wrap up bug 1226379 and then we'll kick off another funnelcake to confirm.

It is just a bit annoying because I ran some funnelcakes in November had data split betweeen FHR and UT (between rolls out as clients were updating to UT) and I was expecting to compare numbers from FHR and UT. Maybe the funnelcake data is there in another field?

The funnelcakes .ini and cfgs normally include values lie:

+aus="funnelcake50"
+dist_id="funnelcake50"
+id=mozilla50
+app.partner.mozilla58="mozilla50"

I am specifically looking for funnelcake 58, 59, 60, 61, and 62 that were kicked off after Nov 1st and some of the data should be in UT.

Could be check all four of those fields in UT to see if any of them are there?

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Where do we get funnelcake builds in order to test them?

Comment 7

[Alessio Placitelli [:Dexter]]

(In reply to Mark Reid [:mreid] from comment #0)
> Funnelcake builds use a special identifier for appUpdateChannel, for
> example, instead of "release", we would see a value such as
> "release-cck-mozilla51" for the channel.

Yes, it used to be like that, before bug 1165304 landed :) Now we're only reporting "release", as the remaining informations are in the "partner" section. See bug 1165304 comment 6.

With the latest "Mozilla EME-Free" (v 44.0.1 / buildid 20160205155049) build, I get the following values in the pings:

partner.distributionId 	mozilla-EMEfree
partner.distributionVersion 	1.0
partner.partnerId 	null
partner.distributor 	null
partner.distributorChannel 	null
partner.partnerNames 	[mozilla-EMEfree]

Comment 8

[Chris More [:cmore]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #6)
> Where do we get funnelcake builds in order to test them?

Releng normally creates them for me. Check out bug 1216740 for an example. We normally ask for them just before the start of a major release and then distribute them a week after the start of a release just in case we had a chemspill.

Comment 9

[Chris More [:cmore]]

(In reply to Alessio Placitelli [:Dexter] from comment #7)
> (In reply to Mark Reid [:mreid] from comment #0)
> > Funnelcake builds use a special identifier for appUpdateChannel, for
> > example, instead of "release", we would see a value such as
> > "release-cck-mozilla51" for the channel.
> 
> Yes, it used to be like that, before bug 1165304 landed :) Now we're only
> reporting "release", as the remaining informations are in the "partner"
> section. See bug 1165304 comment 6.
> 
> With the latest "Mozilla EME-Free" (v 44.0.1 / buildid 20160205155049)
> build, I get the following values in the pings:
> 
> partner.distributionId 	mozilla-EMEfree
> partner.distributionVersion 	1.0
> partner.partnerId 	null
> partner.distributor 	null
> partner.distributorChannel 	null
> partner.partnerNames 	[mozilla-EMEfree]

Ok, well this breaks funnelcake reports now and I wish I would have known this before this change was made. Need to pick the pieces and figure out what to do now.

Comment 10

[Benjamin Smedberg]

cmore, the data is not missing in raw form. Is there a specific derived stream which is missing the partner data which needs to be fixed for the funnelcake reports you're talking about?  This has been the case for so long that it predates the change-notification announcements that we now send to fhr-dev.

Comment 11

[Chris More [:cmore]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #10)
> cmore, the data is not missing in raw form. Is there a specific derived
> stream which is missing the partner data which needs to be fixed for the
> funnelcake reports you're talking about?  This has been the case for so long
> that it predates the change-notification announcements that we now send to
> fhr-dev.

Ok, so in raw form, whatever is being set by the funnelcake configs, is still all there?

We've been reading the funnelcake channel name via the app.update.channel for all of 2015 retention tests and not the partner build field. It looks like we will have to switch to reading the partner value as the release channel will just be release|beta|aurora|nightly. This is probably why mreid is not seeing any of the funnelcake values in the UT retention data.

Comment 12

[Benjamin Smedberg]

It is. I'm going to call this WFM.

Comment 13

[Chris More [:cmore]]

Ok, we'll have :mreid use the partner ID value and I'll see if I can get the partner ID value pulled into other reports where we have historically used the update channel. Thanks

Comment 14

[Mark Reid [:mreid]]

Thanks Benjamin, I've verified that environment.partner.distributionId does appear to include the funnelcake identifiers - it has values of the form 'mozilla51'.  I'll make the required changes to the Churn dataset.

Comment 15

[Dave Zeber [:dzeber]]

(In reply to Chris More [:cmore] from comment #5) 
> I am specifically looking for funnelcake 58, 59, 60, 61, and 62 that were
> kicked off after Nov 1st and some of the data should be in UT.
> 
> Could be check all four of those fields in UT to see if any of them are
> there?

I can confirm that these funnelcakes are all showing up in the UT data from January. These are identified using environment.partner.distributionId where appUpdateChannel = 'release', as discussed above.