651283 - Migrate orphaned users to latest m-c

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

In today's Aurora meeting, RelEng were asked to create updates for orphaned nightly users of:

FF4.2a1pre
FF34.0.13pre 
FF4.0.12pre

...to allow them migrate to latest m-c (6.0a1) build (on nightly channel). This will help build up the nightly user coverage on m-c.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Based on metrics numbers, if we get all of those users to migrate, we will be adding ~15k users to m-c.

Comment 2

[chris hofmann]

pentaho shows around 20k

4.2a1pre=  8,982	
13pre   = 10,255
12pre   =  2,082

I think the lower numbers that daniel passed around this afternoon were from these releases plus additional "nightly" channel info.

I'm wondering if it is possible to have a 4.2a1pre, 13pre, or 12pre version user that is not on the nightly channel, on another channel, or is reporting no channel information?

that would explain the difference in the numbers daniel's channel report passed around this afternoon and latest numbers on pentaho. 

if there are 4.2a1pre, 13pre, or 12pre that aren't on the nightly channel, is is possible to construct the snippets to capture those users too?

Comment 3

[Rail Aliiev [:rail]]

Looks like we already provide proper snippets for 4.2a1pre and it should be upgraded to latest 6.0a1, see

https://aus3.mozilla.org/update/3/Firefox/4.2a1pre/0000000000/Linux_x86_64-gcc3/en-US/nightly/Linux%202.6.38-8-generic%20%28GTK%202.24.4%29/default/default/update.xml?force=1

<?xml version="1.0"?>
<updates>
    <update type="minor" version="6.0a1" extensionVersion="6.0a1" buildID="20110420030555">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011/04/2011-04-20-03-mozilla-central/firefox-6.0a1.en-US.linux-x86_64.complete.mar" hashFunction="sha512" hashValue="a567763a073ca26003d7cd270ebc2f531566da8c1c70023364582b59381105d5a9a224cfefec276a8670292cfe5971f3b8edfd77326560839d6ecf1b070ac9ac" size="15894800"/>
    </update>
</updates>

Comment 4

[Rail Aliiev [:rail]]

Attachment: Add AUS2 exceptions for 4.0.12pre and 4.0.13pre

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 527320 [details] [diff] [review]
Add AUS2 exceptions for 4.0.12pre and 4.0.13pre

$ cvs commit -m "Bug 651283 - Migrate orphaned users to latest m-c. p=rail, r=rhelmer"
Checking in inc/config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.131; previous revision: 1.130
done


$ cvs tag AUS2_PRODUCTION
W inc/config-dist.php : AUS2_PRODUCTION already exists on version 1.130 : NOT MOVING tag to version 1.131

$ cvs tag -F AUS2_PRODUCTION
T inc/config-dist.php

$ cvs tag AUS2_RTM_`date +%Y%m%d%H%M`
(which created AUS2_RTM_201104202159)

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #2)
> pentaho shows around 20k
> 
> 4.2a1pre=  8,982    
> 13pre   = 10,255
> 12pre   =  2,082
When was this data collected? Was this collected, and measured, in the same way as the rest of teh data presented Tuesday?


> I think the lower numbers that daniel passed around this afternoon were from
> these releases plus additional "nightly" channel info.
> 
> I'm wondering if it is possible to have a 4.2a1pre, 13pre, or 12pre version
> user that is not on the nightly channel, on another channel, or is reporting no
> channel information?
> 
> that would explain the difference in the numbers daniel's channel report passed
> around this afternoon and latest numbers on pentaho. 

deinspanjer: can you clarify where the difference in numbers comes from?




> if there are 4.2a1pre, 13pre, or 12pre that aren't on the nightly channel, is
> is possible to construct the snippets to capture those users too?
aiui, 4.2a1pre, 13pre, or 12pre should *only* be on the nightly channel. The 5k users (difference between 20k and 15k) is such a tiny percentage of our ADUs, maybe these are frankenfox, or some error in our measuring? 

If deinspanjer thinks the numbers are accurate, and the cause of the number discrepancy is different channels, we should revisit. It may be possible to figure out what channels they are on, ignore how they got there, and custom-generate snippets to update them back to something normal. Not sure how these custom updates would be QA'd, but lets deal with that if we need to, after we hear from deinspanjer.

Comment 7

[chris hofmann]

> pentaho shows around 20k
> 
> 4.2a1pre=  8,982    
> 13pre   = 10,255
> 12pre   =  2,082
When was this data collected? 

2011-04-19

Was this collected, and measured, in the same way
as the rest of the data presented Tuesday?

the number I put here are from our standard pentaho report 
https://metrics.mozilla.com/pentaho/content/pentaho-cdf/RenderHTML?solution=metrics&path=blocklist/blocklist_0&dashboard=template.html&template=new-metrics&title=Firefox%20Usage

I'm not sure about the source of the build + channel numbers I've never seen that report before.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

Here are updated breakdowns by channel for those three versions.  Note there are another 5 channels reported, but there are only one or two daily pings on them so I left them out to save room:
[Channels].[Unknown], [Channels].[default-cck-opensuse], [Channels].[default-cck-mozillaonline], [Channels].[nightly-cck-yandex], [Channels].[release-cck-yandex]

http://screencast.com/t/i1J558OMZ0AU

Comment 9

[Chris AtLee [:catlee]]

(In reply to comment #8)
> Here are updated breakdowns by channel for those three versions.  Note there
> are another 5 channels reported, but there are only one or two daily pings on
> them so I left them out to save room:
> [Channels].[Unknown], [Channels].[default-cck-opensuse],
> [Channels].[default-cck-mozillaonline], [Channels].[nightly-cck-yandex],
> [Channels].[release-cck-yandex]
> 
> http://screencast.com/t/i1J558OMZ0AU

"default" are probably try builds, or other non-nightly builds.

Comment 10

[chris hofmann]

(In reply to comment #9)
> (In reply to comment #8)
> > Here are updated breakdowns by channel for those three versions.  Note there
> > are another 5 channels reported, but there are only one or two daily pings on
> > them so I left them out to save room:
> > [Channels].[Unknown], [Channels].[default-cck-opensuse],
> > [Channels].[default-cck-mozillaonline], [Channels].[nightly-cck-yandex],
> > [Channels].[release-cck-yandex]
> > 
> > http://screencast.com/t/i1J558OMZ0AU
> 
> "default" are probably try builds, or other non-nightly builds.

874 of these non-nightly channel pings from 4.0b12pre builds, 
and 3635 pings from similar 4.0b13pre builds make the numbers seem very high if this is just try builds.  earlier this month there were 13,000 4.013pre "default channel" users.

the better questions are what are these other possible sources for builds, and maybe even more important are the update snippets catching them?

Comment 11

[Chris AtLee [:catlee]]

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > Here are updated breakdowns by channel for those three versions.  Note there
> > > are another 5 channels reported, but there are only one or two daily pings on
> > > them so I left them out to save room:
> > > [Channels].[Unknown], [Channels].[default-cck-opensuse],
> > > [Channels].[default-cck-mozillaonline], [Channels].[nightly-cck-yandex],
> > > [Channels].[release-cck-yandex]
> > > 
> > > http://screencast.com/t/i1J558OMZ0AU
> > 
> > "default" are probably try builds, or other non-nightly builds.
> 
> 874 of these non-nightly channel pings from 4.0b12pre builds, 
> and 3635 pings from similar 4.0b13pre builds make the numbers seem very high if
> this is just try builds.  earlier this month there were 13,000 4.013pre
> "default channel" users.

We've had people blog about try builds, so that's not too surprising to me.

> the better questions are what are these other possible sources for builds, and
> maybe even more important are the update snippets catching them?

nothing happens ATM for people on "default", they're completely ignored

Comment 12

[Rail Aliiev [:rail]]

I've just checked AUS2 responses and it looks like all nightly users of 4.0.12pre, 4.0.13pre and 4.2a1pre should be able to get updates to the latest mozilla-central based nightlies.

https://aus2.mozilla.org/update/1/Firefox/4.0.13pre/000000000/WINNT_x86-msvc/en-US/nightly/update.xml

https://aus2.mozilla.org/update/1/Firefox/4.0.12pre/000000000/Linux_x86-gcc3/en-US/nightly/update.xml

https://aus2.mozilla.org/update/1/Firefox/4.2a1pre/000000000/Darwin_x86_64-gcc3/en-US/nightly/update.xml

Comment 13

[Rail Aliiev [:rail]]

I've just noticed that there is a possible typo in comment #0

> FF34.0.13pre 
> FF4.0.12pre

Shouldn't they be 4.0b13pre and 4.0b12pre? If yes, the patch should be rewritten.

Comment 14

[LegNeato]

Yes, it should be 4.0b13pre and 4.0b12pre

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Blocklist vs AUS counts

tl;dr: I think we're already offering updates on the nightly channel, and I don't know what more RelEng can do to influence people to update faster. We also have to be careful about how we use our metrics data.

Not the tl;dr:
* The application versions of interest are 4.0b12pre, 4.0b13pre, 4.2a1pre; which should be offered 6.0a1 (ie current mozilla-central tip)
* I agree with Rail what we're already providing those paths for the *nightly* channel
* Also agree that attachment 527320 [details] [diff] [review] is faulty. I'd suggest reverting it

* Try builds are actually on the nightly channel, so people get updated back to mozilla-central (whether they want to or not)
* Builds end up on the default channel when there's not a 
   ac_add_options --enable-update-channel=<something>
line in the mozconfig, which means people doing their own builds, and any distributor not aware of how this stuff works
* The updater is enabled by default in Firefox builds, but (eg) linux distributions will pretty reliably disable updates because they have their own package systems
* In this scenario the build would not ping AUS, but would hit blocklist on the default channel

* The ADU in comment #8 are from blocklist data, which I can reproduce. Switching it to AUS data results in higher counts, up ~50% for 4.2a1pre, while pretty much suppressing the pings on default (see this attachment). This represents the users actually pinging us for updates, the people we can reach
* I'm not sure I believe higher AUS pings, but we do query every 8 hours for nightly builds, instead of every 24 hours (if the app is open, of course). Dunno if metrics are somehow crunching that to give unique users, or if it's just the raw number of pings. Daniel ?

* The platform and locale distributions for the 'orphaned' users are broadly consistent with unorphaned ones. So we're not leaving some platforms behind, and the vast majority are windows users - about a 50%/50% split between XP and Windows 7. (based on AUS data)

* The ADU in comment #8 do show the blocklist counts are dropping, by 50% for 4.0b13pre over 20 days, and by 80% for 4.2a1pre in the 7 days since 2011-04-13 (version change). I'm inclined to believe that people are using the updater, rather than manually upgrading using an installer. So the problem boils down to the counts not dropping fast enough

Questions:
* How do the count decays vs time compare to version changes for nightlies builds of 3.5/3.6/4.0 branches ? Have we regressed the 'punctuality' of the updater ?

* Does the updater give up on downloading an update if there's a newer one available ? I don't think so, but worth asking rs. For users that don't run Firefox for long periods, and do background updates, this could easily result in downloading only a portion of a complete update, then throwing that away for the next one.

Comment 16

[Daniel Einspanjer [:dre] [:deinspanjer]]

There isn't any magic we can do to de-dupe AUS pings. Also the Linux distro problem is an important reason why we switched to using blocklist pings to track.

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 527320 [details] [diff] [review]
Add AUS2 exceptions for 4.0.12pre and 4.0.13pre

(In reply to comment #15)
> * Also agree that attachment 527320 [details] [diff] [review] is faulty. I'd suggest reverting it

Reverted:

$ cvs commit -m "Bug 651283 - Migrate orphaned users to latest m-c. Reverted revision 1.131."
cvs commit: Examining .
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.134; previous revision: 1.133
done

$ cvs -q tag AUS2_PRODUCTION
W inc/config-dist.php : AUS2_PRODUCTION already exists on version 1.133 : NOT MOVING tag to version 1.134

$ cvs -q tag -F AUS2_PRODUCTION
T inc/config-dist.php

$ cvs tag AUS2_RTM_`date +%Y%m%d%H%M`
(which was AUS2_RTM_201104260458)

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #15)

> * Does the updater give up on downloading an update if there's a newer one
> available ? I don't think so, but worth asking rs. For users that don't run
> Firefox for long periods, and do background updates, this could easily result
> in downloading only a portion of a complete update, then throwing that away for
> the next one.

rstrong: can you help clarify how updater works in this scenario?

Comment 19

[Robert Strong (they/them - no direct email)]

(In reply to comment #15)

> * Does the updater give up on downloading an update if there's a newer one
> available ? I don't think so, but worth asking rs. For users that don't run
> Firefox for long periods, and do background updates, this could easily result
> in downloading only a portion of a complete update, then throwing that away for
> the next one.
You are correct. It doesn't give up on the download of the update when a newer one is available.

Comment 20

[Robert Strong (they/them - no direct email)]

We *might* want to override extension compatibility checks for these nightly users by setting the extensionVersion in the snippet to the same version as the Firefox being offered the update which will cause the update to be downloaded in the background.

Comment 21

[Robert Strong (they/them - no direct email)]

Might be a good thing to offer updates to the remaining users from bug 518508 in this bug if they aren't being offered updates at this time.

Comment 22

[Rail Aliiev [:rail]]

Back to the pool. I can grab it again if we have some clear scenario here.

Comment 23

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Per RelEng meeting, these updates are already in place, and being offered to users of older nightlies. Nothing left for us to do here, so closing as WFM. 

Please reopen with details if there's anything for us to do that I missed.