518508 - Update stranded "4.0a1pre" users to a valid nightly

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Daniel Veditz [:dveditz]]

For a short while after the 1.9.0 branch was cut "Minefield" nightlies were called 4.0a1pre. The trunk was later renumbered 3.6a1pre and stranded users who did not manually download a new version since client updates will not down-rev people. We currently appear to have about 7K of these stranded users, about the same number as those using the active trunk 3.7a1pre Minefield builds and nearly double the number using the latest Namoroka nightlies.

We need to prepare update snippets to move these people onto a supported nightly, both for their security and to help them fulfill their apparent intention to help us test.

To which build do we direct them? Beltzner says trunk nightlies because that's what they were originally using. I argue Namoroka nightlies because that's where we most need the help testing, and I believe that matches what happened to then-trunk nightly testers when we cut the 1.9.2 branch. Beltzner wins if he doesn't buy my arguments.

Either way the update snippets will have to say some version higher than what they've currently got (e.g. "4.0a2pre") but can point at an existing complete mar update file. We'll have to test that of course, but should work in theory.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The number of people we can actually reach is probably much less than 7K, more like 2K. Still worth looking into though.

If you dig into the AUS metrics, limiting to Product=Firefox, Version=4.0a1pre, and averaging over Sept 1 to Sept 9, then there are 
7813 users. Of those 5582 (71%) are on the default channel, and the rest on the nightly channel. The most common buildID's are
nightly:   
  2008050802    197
  2008051003   1758
  2008051302    134
default:
  2008031002   1233
  2008031602    683
  2008041102   2803
The first group may be afflicted by a problem with the updater, I'll check if they're stuck or not. If the updater was working then I'd expect almost all of them to be on the same build, because we stopped offering updates to 4.0* users long after the version was reduced.

For the "default" builds, the channel-prefs.js is setting the channel to default, so that was an omission in the mozconfig (not specifying a channel and so falling back to the default in the build system). Not sure we can convince AUS to talk to them without affecting other people who compiled their own builds.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> The number of people we can actually reach is probably much less than 7K, more
> like 2K. Still worth looking into though.

Given how small this group is, can you clarify the importance of this? Is there something specifically important about these orphaned 4.0*pre users? Otherwise, could we just reopen bug#383676 and deal with these like other orphaned users, as time permits?

Comment 3

[Johnathan Nightingale [:johnath]]

This group is a nice test-case for the work that we will eventually want to do for other stranded groups. It's a single version ID, there's zero risk of them *wanting* to stay there for some reason (since the version doesn't correspond to a specific set of functionality, just a point-in-time naming glitch - people 'orphaned' on 2.0.0.x or even 3.0.8 might be intentionally staying there for some reason) and my hope is that it would be an easier problem to solve than "recover all orphaned users."

Is that helpful, John? Do you foresee this being a significant amount of work, where we should be careful to pick only the highest value target? Or is this reasonable as a scoped-down trial balloon?

Comment 4

[alice nodelman [:alice] [:anode]]

Looks like no one is commenting here, last question was for joduinn.

joduinn - does this look like something that we want to work on soon?

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Daniel,  do we have some current numbers for this category of user?

Comment 6

[Aki Sasaki (not active)]

Assigning to joduinn for prioritization.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> Is that helpful, John? Do you foresee this being a significant amount of work,
> where we should be careful to pick only the highest value target? Or is this
> reasonable as a scoped-down trial balloon?
Scoping down the problem is an excellent reality-check, Johnauth, thanks. 

It feels like we should focus effort on where the most users are, so I'd like to identify the largest clump of orphaned users, and directly try to upgrade those. By *far*, the biggest clump is on FF2.0.0.20, several orders of magnitude more then are on FF4.0a1. Therefore, as we discussed, I'm going to close this bug as WONTFIX, and have instead opened bug#526409 to track a new MU offer for those users on FF2.0.0.20. 

Once we do that, we can reevaluate doing another upgrade for the next largest clump of orphaned users.

Let me know if I missed anything, ok?

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #7)
> (In reply to comment #3)
> > Is that helpful, John? Do you foresee this being a significant amount of work,
> > where we should be careful to pick only the highest value target? Or is this
> > reasonable as a scoped-down trial balloon?
> Scoping down the problem is an excellent reality-check, Johnauth, thanks. 
Typo. Of course, that should be "Johnath". Sorry!

Comment 9

[Jesse Ruderman]

It's true that this is a small number of users, but they're users who want to be nightly users, so they're especially valuable.

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

Re-opening as we fixed bug 526409 a while ago, and this bug has yet to be resolved. Current ADUs show as many (if not more) people using Firefox 4.0a1pre as using Firefox 3.7a5pre (about 14,000) across all channels. Nick mentions that only about half of those users are on the nightly channel, meaning that some are compiling it themselves (which is odd enough).

Anyway - I think we should do an across-all channels unadvertised update that takes:

Firefox 4.0a1pre -> latest nightly

and also moves them onto a nightly channel.

Comment 11

[Benjamin Smedberg]

I think the "compiling it yourself" thing has got to be a red herring: self-compiled builds will only identify as 4.0a1pre if they were compiled between 14-Feb-2008 and 13-May-2008.

I still don't understand where this data can be coming from, though: is this blocklist/AMO data, or AUS data? The metrics dashboard is really confusing to me.

Comment 12

[Mike Beltzner [:beltzner, not reading bugmail]]

I was using blocklist pings

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Repeating the data mining I did earlier (comment #1) - using AUS data, limiting to Product=Firefox, Version=4.0a1pre, and reporting the average number of requests over the week ending 2010-04-19. There are about 7700 users on nightly, 7800 on default. The most common buildID's are
nightly:   
  2008050802    895
  2008051003   6607
default:
  2008031002   1318
  2008031602   1500 
  2008041102   4158

These are all en-US Windows users, about 80% using XP but about 10% each for Vista & 7. There are no buildID's after 2008060602 so I agree with bsmedberg that this is not people compiling themselves (at least not any time recently). Perhaps the increase over time is from some blogs/websites out there pointing to the old builds, basically getting excited that 4.0 > 3.x.

The group on nightly we can probably get to easily, in fact I hacked up a test and we don't even need to fake the version to keep the updater happy. For those on default we'll need the AUS changes in bug 534954 (in order to be able to update only default users with version 4.0a1pre) and a custom mar file to update channel-prefs.js. I'd rather do this all at once so I'm going to block on 534954.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] AUS changes on top of bug 534954

This worked for me to test both nightly and default cases, after setting up a mozilla-central-catchup/ as a subset of mozilla-central/ (last two en-US dirs for WINNT_x86-msvc). For realz we'd use a snippet pointing to the custom mar, and those users would go
    old build --> latest build when mar created --> latest build

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 440253 [details] [diff] [review]
[cvs] AUS changes on top of bug 534954

Config style changed in bug 534954. Will need to rethink this.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Update 4.0a1pre/nightly users to mozilla-central

This gets us the set of 4.0a1pre users using the nightly channel. Give them mozilla-central for now, but they'll get mozilla-1.9.3 when that branch is cut.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

For 4.0a1pre/default, we can address these builds by creating snippets in the release side of the snippet store (ie Firefox/4.0a1pre/BUILD_TARGET/BUILDID/default/). We need an explicit list of buildID's for that, these are the ones with counts >=10 over the last week, which happen to all be WINNT_x86-msvc: 
2008030702
2008030802
2008031002
2008031102
2008031402
2008031502
2008031602
2008031702
2008031802
2008040802
2008041102

Then we create a custom mar that updates channel-prefs.js to use nightly. After getting that update they'll go the same path as 4.0a1pre/nightly users.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

On aus2-staging:

cd /opt/aus2/snippets/staging
mkdir 20100530-4.0a1-on-default && cd $_
mkdir -p Firefox/4.0a1pre/WINNT_x86-msvc/2008041102/en-US/default
cd Firefox/4.0a1pre/WINNT_x86-msvc/
for d in 2008030702 2008030802 2008031002 2008031102 2008031402 2008031502 \
         2008031602 2008031702 2008031802 2008040802; do \
  ln -s 2008041102 $d
done
cd 2008041102/en-US/default
cat > complete.txt << EOF
version=1
type=complete
url=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/bug518508-update-channel/update-channel.mar
hashFunction=SHA1
hashValue=295dd63ff1e41ce0bc5322d19b932b0200c250f0
size=311    
build=2010053000
appv=4.0a1pre
extv=4.0a1pre
EOF

This is ready to go, look OK Ben ? The mar file is owned by ffxbld so it won't get killed out of there in 90 days.

Comment 19

[Michael Morgan [:morgamic]]

Comment on attachment 448316 [details] [diff] [review]
Update 4.0a1pre/nightly users to mozilla-central

wfm

Comment 20

[[e-mail] (:bhearsum)]

> This is ready to go, look OK Ben ? The mar file is owned by ffxbld so it won't
> get killed out of there in 90 days.

Looks good to me.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #18)
On aus2-staging:

$ ~/bin/backupsnip/20100530-4.0a1-on-default
$ ~/bin/pushsnip/20100530-4.0a1-on-default 

So 4.0a1pre users on default channel will be repatriated to nightly now.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 448316 [details] [diff] [review]
Update 4.0a1pre/nightly users to mozilla-central

Checking in inc/config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.80; previous revision: 1.79
done

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Tagged and deployed in bug 569160.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Still 3k users on default with buildID's like those in comment #17. The mar file had been removed from ftp.m.o by cron'd clean up jobs, restored it with permissions which make that impossible.

Comment 25

[Robert Strong (they/them - no direct email)]

Nick, just wanted to verify that the mar is still being offered?

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Should be, eg ('scuse the old school url format)
https://aus2.mozilla.org/update/1/Firefox/4.0a1pre/2008041102/WINNT_x86-msvc/en-US/default/update.xml
That'd get them on nightly then up to latest on the next update.

There would have been some period between comment 23 and comment 24 when it wasn't offered but I don't how long that would have been.

Comment 27

[[e-mail] (:bhearsum)]

I couldn't find any users at all on 4.0a1pre, and the MAR file from here was causing an error in one of our cleanup cronjobs, so I moved it away to ~ffxbld.

Comment 28

[Robert Strong (they/them - no direct email)]

In the last 30 days there have been 88358 AUS pings (not blocklist pings) for Firefox version 4.0a1pre. All of them were en-US except for 2 pings for zh-CN. That equates to over 2900 users if each one launched Firefox 4.0a1pre every day which is highly unlikely so the number of users is quite likely higher.

Comment 29

[Robert Strong (they/them - no direct email)]

Forgot to mention that a link to the data is in bug 651779.

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

The current state is that the update offer is up (example url in comment #26) but it points to a channel switching mar which is a 404. I can probably find a copy of the mar, but if they haven't updated while the path has been live for several months what hope can we have they ever will ?

Comment 31

[Robert Strong (they/them - no direct email)]

We've seen slow uptake where it takes months for the stragglers to update previously and I think it is best to always have the update available for these users so they can update when the mood strikes them.

Comment 32

[Robert Strong (they/them - no direct email)]

I also think we should just advertise this update with 4.0a1pre for the extensionVersion so it will by default not prompt the user. The numbers I provided in comment #28 are for AUS pings so chances are the majority of those users would just get updated.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

I'm much less optimistic that users who haven't updated in months ever will, but as long as the number of these hacks is limited it's OK they live on.

To fix up people still stuck on default I put the mar file back and added a cron job for ffxbld@stage so that it doesn't go away again.

@weekly   touch /pub/mozilla.org/firefox/nightly/experimental/bug518508-update-channel/update-channel.mar

(In reply to comment #32)
> I also think we should just advertise this update with 4.0a1pre for the
> extensionVersion so it will by default not prompt the user. 

We are already doing this for people on default, which delving in the metrics is roughly half of your 4.0a1pre count. If we want to do this on the nightly channel then then bug 651283, or a new bug is where we should do that. This one is complicated enough already.

Comment 34

[Robert Strong (they/them - no direct email)]

Thanks Nick!