651982 - Create custom snippets to migrate FF3.7a{3,4,5} and FF4.0b* users to mozilla-beta

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

To start growing our user base on mozilla-aurora, we'd been asked to migrate beta users from: FF3.7a*, FF4.0b* over to mozilla-aurora.

We need to explicitly not migrate any beta users on FF3.5.x, FF3.6.x, FF4.0.x, as they are beta-users for supported releases, and we need them there.

Comment 1

[LegNeato]

To be clear:
  FF3.7a*, FF4.0b*, and _FF4.0_ on the beta channel should get the update (the FF4 users were left out of the comment above)

Comment 2

[Mike Taylor [:bear]]

FYI may be helpful as it's doing something similar: 
 Bug 651283 - Migrate orphaned users to latest m-c

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Change of plan during yesterday's meeting, morphing this bug to match: 

1) Instead of moving orphaned beta users to mozilla-aurora, we're moving them to FF5.0b1.

2) Here's the users we're reaching to, based on blocklist counting with nthomas, of beta channel users, en-US only, using data from 26apr:
4.0b12 65k
4.0b11 76k
4.0b10 43k
4.0b9 36k
4.0b8 46k
4.0b7 51k
4.0b6 23k
4.0b5 6k
4.0b4 13k 
4.0b3 11k
4.0b2 19k
4.0b1 26k
..and for 3.7a1...5, there is a combined total of 2.4k

3) We will offer prompted MUs to these specific beta users, and none others, to upgrade to FF5.0b1. Once we get ~100k users on FF5.0b1, we will throttle this MU, and wait-and-see if there are problems with FF5.0b1. 

4) The hope is that some of the FF5.0b1 users would self-select to channel switch to aurora. If many do this, we can unthrottle the MU some more, to repopulate the FF5.0b1 pool back up to ~100k.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: major update configs for updating to 5.0b1 with 64bit platforms included

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: update configs for 5.0b1, removed RCs for major update

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 528931 [details] [diff] [review]
major update configs for updating to 5.0b1 with 64bit platforms included

r+ in bug 652641

Comment 7

[Chris AtLee [:catlee]]

Attachment: mozilla-2.0 major update configs

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 528939 [details]
mozilla-2.0 major update configs

4.0.1 is in there, that's probably not a good idea :)

Comment 9

[Chris AtLee [:catlee]]

Attachment: mozilla-2.0 major update configs

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 528942 [details]
mozilla-2.0 major update configs

looks good to my rather untrained eyes.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 528942 [details]
mozilla-2.0 major update configs

Overall this compares favourably with moz20-branch-patcher2.cfg, but there are a few issues to fix:

>            <complete>
>                betatest-url   http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/5.0b1-candidates/build1/update/%platform%/%locale%/firefox-5.0b1.complete.mar
>                path   firefox/nightly/5.0b1-candidates/build1/update/%platform%/%locale%/firefox-5.0b1.complete.mar
>                url   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/5.0b1-candidates/build1/update/%platform%/%locale%/firefox-5.0b1.complete.mar
>            </complete>

We need to have a <partial> block that is a copy of this, otherwise patcher will complain. 

I think we could just remove betatest-url, so that betatest is the same as beta, and not need releasetest at all. But if you'd rather keep everything the same as other configs that's fine. (ftp.m.o is now guarded by a zeus cache, and stage-old = stage = surf and isn't, so there's at least potential for caching issues).

>            testchannel   betatest releasetest

We don't strictly need releasetest here

>            <5.0b1>
>                extension-version   5.0
>                prettyVersion   5.0
>                version   5.0b1

Nicely done!

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 528932 [details] [diff] [review]
update configs for 5.0b1, removed RCs for major update

I think we should copy these files like so
  moz20-firefox-linux.cfg   --> moz20-firefox-linux-major.cfg
  moz20-firefox-linux64.cfg --> moz20-firefox-linux64-major.cfg
  moz20-firefox-mac64.cfg   --> moz20-firefox-mac64-major.cfg
  moz20-firefox-win32.cfg   --> moz20-firefox-win32-major.cfg
before making this modification. That way we can do any 4.0.2 without having to undo this.

And the patcher config is to be moz20-branch-major-update-patcher2.cfg ?

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 528932 [details] [diff] [review]
update configs for 5.0b1, removed RCs for major update

Also,
* the list of locales needs to be trimmed to en-US to keep the log output useful
* we should do a full check from 4.0b12 to 5.0b1, which means define from and to. We also need the aus_server and ftp_server definitions too

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] patcher config

Differences from attachment 528942 [details]
* added partial block
* left betatest-url in
* removed releasetest from testchannel list
* set locale lists to en-US for all releases (this will need undoing later)

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Add verify configs

Differences from bug 528932
* hg cp first
* set up a full check from 4.0b12 -> 5.0b1 (apply and diff), and snippet check for everything else
* trim locale lists to 'en-US' to avoid spurious errors in logs (will need to be undone later)

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

That patch looks pretty hideous, but actually isn't. It also changes the build target for mac to 32 Intel builds.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] patcher config (that works!)

'path' needs to be unique filenames to avoid partial generation, and make the releasetest usage consistent. This generates sensible looking snippets.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Add verify configs (with better mac build targets)

Same as attachment 529016 [details] [diff] [review], except it uses better build targets for macs and adds 3.7a1 thru 3.7a4.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

This describes my steps
  https://wiki.mozilla.org/Releases/Firefox_5.0b1/BuildNotes#Updates
Note that I've used UPDATE_PACKAGING_R13 to get good things like only complete snippets, and useful mac build targets (for b7 onwards anyway).

Snippets have been generated and pushed on betatest and releasetest. Update verify has not been run yet. The billboard isn't up yet (bug 653636), which may be an issue for QA doing testing. Over to lsblakk/catlee.

Comment 20

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

lsblakk, nthomas: Per discussion with rstrong in corridor, we should *not* offer MU to FF3.7a1, FF3.7a2 users. They have a different size billboard. FF3.7a3 and above, as well as FF4.0b*, are all still ok to MU.

At a later time, we need to revisit how to deal with the FF3.7a1, FF3.7a2 users, but not in this current rush, and not in this bug. There is also very few people on FF3.7a1, FF3.7a2 so it is not a problem.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Moved the snippets for 3.7a1 and 3.7a2 so that they are not offered.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] patcher config (after tweaks)

This changes the detailsUrl (../5.0beta/details/), and removes updates for 3.7a1 and 3.7a2. There's still a bunch of manual post processing that needs doing (remove partials for 4.0b12, fix mac build targets ...).

Carrying review.
Checking in moz20-branch-major-update-patcher2.cfg;
/cvsroot/mozilla/tools/patcher-configs/moz20-branch-major-update-patcher2.cfg,v  <--  moz20-branch-major-update-patcher2.cfg
initial revision: 1.1

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Add verify configs (drop 3.7a1/a2)

Same as reviewed patch only I dropped the checks on 3.7a1 and 3.7a2.

http://hg.mozilla.org/build/tools/rev/ef8535325065

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529418 [details] [diff] [review]
[tools] Add verify configs (drop 3.7a1/a2)

Follow up fix to remove trailing spaces on mac 4.0b5 and 4.0b6 build targets:
 http://hg.mozilla.org/build/tools/rev/10deea32888a

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] fix mac buildIDs in patcher config

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Fix buildIDs for mac for update verify

Comment 27

[Robert Strong (they/them - no direct email)]

Just wanted to check what builds aren't going to be updated in case QA runs into another one like bug 654343.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #27)

No updates for:
* Any 3.7a1 or 3.7a2 build, any 4.0rc or 4.0.1 build
* Any Mac 64bit mono-arch build (although we do have test snippets up)
* Any PPC Mac build
* Any candidate build that was superseded by a subsequent candidate build (ie not a build1 if build2 shipped)

Comment 29

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: throttling 5.0 to 50%

joduinn mentioned that we should throttle the update and keep an eye on it so I added 5.0 to throttling at 50%

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529788 [details] [diff] [review]
throttling 5.0 to 50%

Looks fine.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529621 [details] [diff] [review]
[cvs] fix mac buildIDs in patcher config

Checking in moz20-branch-major-update-patcher2.cfg;
new revision: 1.2; previous revision: 1.1

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529622 [details] [diff] [review]
[tools] Fix buildIDs for mac for update verify

http://hg.mozilla.org/build/tools/rev/f7d0963f9601

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529788 [details] [diff] [review]
throttling 5.0 to 50%

Landed this then realized it's not going to work, because the versions in the array have to be the starting points of the update path.

Landing: new revision: 1.137; previous revision: 1.136
Backout: new revision: 1.138; previous revision: 1.137

$ cvs diff -r 1.136 -r 1.138 config-dist.php
<no output>

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Throttle 3.7a3-4.0b12 at 50%

Is not nice. Checked this works for 4.0b11/b12 on aus2-staging.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 529894 [details] [diff] [review]
[cvs] Throttle 3.7a3-4.0b12 at 50%

Switching review to someone in EDT.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Throttle 3.7a3-4.0b12 at 0%

Stop all updates to 5.0b1 on beta channel, once we get to our initial threshold of users.

Comment 37

[Chris AtLee [:catlee]]

Comment on attachment 529894 [details] [diff] [review]
[cvs] Throttle 3.7a3-4.0b12 at 50%

cvs commit: Examining .
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.139; previous revision: 1.138
done

cvs tag -F AUS2_PRODUCTION config-dist.php
T config-dist.php

Comment 38

[Lukas Blakk [:lsblakk] use ?needinfo]

These snippets are now live, resolving.

Comment 39

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Need to keep this open for a bit longer. 

Once we reach 100K migrated users, we'll throttle back and watch for any problems, before we fully unthrottle and migrate the remaining ~300k users over. 

nthomas patch is already r+ for throttling back, just needs to land in production when we reach 100k.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Throttle 3.7a3-4.0b12 at 0% (unrotted)

Unrotted for the changes the 4.0.1 major update made. Carrying review over.

Comment 41

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

lsblakk, nthomas:

1) All looks ok on crash-stats, so lets change from 50% throttled to full 100% unthrottled.

2) Given today's evaluation that everything still looks good on crash-stats, we no longer need to stop at 100K and reevaluate. In today's beta/aurora meeting, we're happy with the results so far, and see no need to stop at 100K. Lets continue and offer FF5.0b1 to all 417K beta users (in comment#3, comment#28).

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Unthrottle 3.7a3-4.0b12

Removes throttling of the update to 5.0b1, as requested.

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 531143 [details] [diff] [review]
[cvs] Unthrottle 3.7a3-4.0b12

Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.141; previous revision: 1.140
done

$ cvs tag AUS2_PRODUCTION config-dist.php
W config-dist.php : AUS2_PRODUCTION already exists on version 1.140 : NOT MOVING tag to version 1.141
$ cvs tag -F AUS2_PRODUCTION config-dist.php
T config-dist.php

Comment 44

[Nick Thomas [:nthomas] (UTC+12)]

Updates to 5.0b1 are unthrottled now.