1412962 - Tracking bug for 2018-01-16 migration work

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

+++ This bug was initially created as a clone of Bug #1407602 +++

The big day is coming. Prepping 58 migration bug in advance to track all special requirements.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We should merge https://github.com/mozilla-releng/releasewarrior-2.0/pull/31 before creating the Fennec 58.0 release in release warrior. I'd create a preflight issue, but it's a bit chicken and egg.

Comment 2

[Jordan Lund (:jlund)]

We also need to land whatever is needed so that we don't bump (and tag?) m-c when we migrate.

There is a patch in works from: Bug 1407602 - Tracking bug for 2017-11-14 migration work

mozreview here: https://reviewboard.mozilla.org/r/195436/diff/1#index_header



I think the past two releases we just manually commented bits out. Something similar to:

@@ -165,12 +165,12 @@
         return self.gecko_repos

     def query_commit_dirs(self):
         dirs = self.query_abs_dirs()
         commit_dirs = [dirs['abs_to_dir']]
-        # if self.config['migration_behavior'] == 'central_to_beta':
-        #     commit_dirs.append(dirs['abs_from_dir'])
+        if self.config['migration_behavior'] == 'central_to_beta':
+            commit_dirs.append(dirs['abs_from_dir'])
         return commit_dirs

     def query_commit_message(self):
         return "Update configs. IGNORE BROKEN CHANGESETS CLOSED TREE NO BUG a=release ba=release"

@@ -324,21 +324,21 @@
             """
         dirs = self.query_abs_dirs()
         next_mb_version = self.get_version(dirs['abs_to_dir'])[0]
         self.bump_version(dirs['abs_to_dir'], next_mb_version, next_mb_version, "a1", "", use_config_suffix=True)
         self.apply_replacements()
-        # # bump m-c version
-        # curr_mc_version = self.get_version(dirs['abs_from_dir'])[0]
-        # next_mc_version = str(int(curr_mc_version) + 1)
-        # self.bump_version(
-        #     dirs['abs_from_dir'], curr_mc_version, next_mc_version, "a1", "a1",
-        #     bump_major=True,
-        #     use_config_suffix=False
-        # )
-        # # touch clobber files
-        # self.touch_clobber_file(dirs['abs_from_dir'])
-        # self.touch_clobber_file(dirs['abs_to_dir'])
+        # bump m-c version
+        curr_mc_version = self.get_version(dirs['abs_from_dir'])[0]
+        next_mc_version = str(int(curr_mc_version) + 1)
+        self.bump_version(
+            dirs['abs_from_dir'], curr_mc_version, next_mc_version, "a1", "a1",
+            bump_major=True,
+            use_config_suffix=False
+        )
+        # touch clobber files
+        self.touch_clobber_file(dirs['abs_from_dir'])
+        self.touch_clobber_file(dirs['abs_to_dir'])


     def beta_to_release(self, *args, **kwargs):
         """ mozilla-beta -> mozilla-release behavior.

@@ -461,14 +461,14 @@
         from_fx_major_version = self.get_version(dirs['abs_from_dir'])[0]
         to_fx_major_version = self.get_version(dirs['abs_to_dir'])[0]
         base_from_rev = self.query_from_revision()
         base_to_rev = self.query_to_revision()
         base_tag = self.config['base_tag'] % {'major_version': from_fx_major_version}
-        # self.hg_tag(
-        #     dirs['abs_from_dir'], base_tag, user=self.config['hg_user'],
-        #     revision=base_from_rev,
-        # )
+        self.hg_tag(
+            dirs['abs_from_dir'], base_tag, user=self.config['hg_user'],
+            revision=base_from_rev,
+        )
         new_from_rev = self.query_from_revision()
         self.info("New revision %s" % new_from_rev)
         pull_revision = None
         if not self.config.get("pull_all_branches"):
             pull_revision = new_from_rev


@sfraser, @callek, does the last instance we used to migrate concur with this? Did we use the instance last time?

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

I don't remember this bit, but the code looks like it'll work, so go for it.

Comment 4

[Johan Lorenzo [:jlorenzo]]

We did use the instance last time. I dug up bugzilla and IRC, and I didn't put too much details in there. The things I did document though are the bugs I hit along the way. In other words, using the instance was pretty smooth :) 

Let me document what I did to use Kim's instance.

1. Read https://github.com/mozilla/releasewarrior/blob/108bd1ff0a5840a412500efa95ee1a9b8a996a6b/how-tos/merge-and-staging-instance.md
2. Decrypt the SSH, located at keys/aws-staging-mergeday.pem.gpg, in the usual location of shared keys.
3. ssh -i aws-staging-mergeday.pem [e-mail] # Note that the connection works fine.
4. scp your SSH *public* key onto [e-mail]:~/$USER.pub # Replace $USER
5. Add -A to command 3. and reuse it
6. Edit ~/.hgrc to change make sure this config exists
> [ui]
> ssh = ssh -C -i ~/.ssh/$USER_rsa.pub # Replace $USER
7. m-b to m-r merge data has lived in ~/merge_day, and m-c to m-c in ~/merge_day_m-c_to_m-b. Feel free to tear these folders apart at will. I do recommend to use 2 different folders, though.


Finally, regarding the push, Rail came up with a simpler way to make sure we don't touch m-c at all (bug 1407602 comment 14). You just have to replace [1] by:
> return (dirs.get('abs_to_dir'),)  # make sure this remains a tuple

That's the patch Rail and I used during the merge.

Please let me know if you have any other questions.

[1] https://dxr.mozilla.org/mozilla-central/rev/ca379fcca95b1f4a3744242ea8647004b99b3507/testing/mozharness/scripts/merge_day/gecko_migration.py#182

Comment 5

[Johan Lorenzo [:jlorenzo]]

Regarding the bugs hit last time, please be aware you may encounter: bug 1415233 and bug 1414278.

The first one will delay each migration by 20 minutes. You will have to run the push a second time in order to make a server-side hook pass.
The second one prevents you from upgrading the wiki automatically.

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: version bump for release

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: version bump for other repos

Comment 8

[Justin Wood (:Callek)]

Comment on attachment 8941094 [details]
version bump for other repos

See comment I'm putting on other patch

Comment 9

[Justin Wood (:Callek)]

Comment on attachment 8941093 [details]
version bump for release

Actually r-, because I'm pretty sure we're doing a central->beta merge early (this week) to prep for a deved b1 for monday, and then the other merges next week.

Comment 10

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #9)
> Comment on attachment 8941093 [details]
> version bump for release
> 
> Actually r-, because I'm pretty sure we're doing a central->beta merge early
> (this week) to prep for a deved b1 for monday, and then the other merges
> next week.

Tom, is TB following suit, and using beta as 59 as of friday, or is there another beta 58 expected before the main merge day?

Comment 11

[Francesco Lodolo [:flod]]

Not sure if it needs to be reported, but we'll have a new language (ia) in Beta 59, from bug 1427688.

Comment 12

[Tom Prince [:tomprince]]

(In reply to Justin Wood (:Callek) from comment #10)
> (In reply to Justin Wood (:Callek) from comment #9)
> > Comment on attachment 8941093 [details]
> > version bump for release
> > 
> > Actually r-, because I'm pretty sure we're doing a central->beta merge early
> > (this week) to prep for a deved b1 for monday, and then the other merges
> > next week.
> 
> Tom, is TB following suit, and using beta as 59 as of friday, or is there
> another beta 58 expected before the main merge day?

We don't have any plans for another 58 beta.

Comment 13

[Justin Wood (:Callek)]

(In reply to Francesco Lodolo [:flod] from comment #11)
> Not sure if it needs to be reported, but we'll have a new language (ia) in
> Beta 59, from bug 1427688.

I added a note on releng's side to validate that this locale is indeed built and ready to ship.

Comment 14

[Jordan Lund (:jlund)]

Attachment: bbot-cfgs_merge_day_gecko_version_mb_mr.patch

this is to be landed before we merge.

Comment 15

[Jordan Lund (:jlund)]

Attachment: bbot-cfgs_merge_day_gecko_version_mc.patch

this is to be landed sometime next week when we bump m-c gecko version.

We should update the docs since this is the new norm.

@tomprince - looking at hg history, we updated comm-beta when we updated mozilla-central last time. I would have thought it be bumped with m-b. Because of my uncertainty, I'm leaving the comm-beta bump out of my patches and will let you put up patches that you think are needed with expected landing dates.

Comment 16

[Jordan Lund (:jlund)]

Attachment: gecko_rm_l10n_nightly_replacements_from_merge_config.patch

This is needed along with the diff mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1412962#c2 *before* we do the migration.

The difference with diff in comment 2 is this can land permanently.

Reason for that this attachment is:

jlund> Jordan Lund I think aki's tip applies here too. we started sourcing a nightly mozconfig and now that is confusing our m-b white/black listing
19:00:05 https://hg.mozilla.org/releases/mozilla-beta/diff/958eef714e2b/mobile/android/config/mozconfigs/android-api-16/l10n-nightly
19:00:57 our merge logic expects this to be here: https://hg.mozilla.org/releases/mozilla-beta/diff/958eef714e2b/mobile/android/config/mozconfigs/android-api-16/l10n-nightly#l1.36

jlund> Jordan Lund I think this is a matter of pointing replacements to the nightly file: https://hg.mozilla.org/releases/mozilla-beta/file/c815231fbfbc/mobile/android/config/mozconfigs/android-api-16/nightly#l14
19:27:25 actually, we can just remove it from the list like ak.i mentioned
19:27:34 since we already change nightly: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/merge_day/beta_to_release.py#18

Comment 17

[Jordan Lund (:jlund)]

callek> oook, I'm running a central_to_beta no-op locally right now, apparantly the /mnt still wasn't big enough on the remote instance (got no space on device errors) -- I wonder if that error was more inode space than disk space though (given it said only ~50% full in `df -h`)

@sfraser - ^ heads up. we may need to clear or add space on the remote machine tomrorow.

Comment 18

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #16)
> Created attachment 8941706 [details] [diff] [review]
> gecko_rm_l10n_nightly_replacements_from_merge_config.patch
> 
> This is needed along with the diff mentioned in
> https://bugzilla.mozilla.org/show_bug.cgi?id=1412962#c2 *before* we do the
> migration.


FTR - this is needed for the m-b->m-r run. I tested m-c->m-b locally and it was successful as is.

Comment 19

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8941706 [details] [diff] [review]
gecko_rm_l10n_nightly_replacements_from_merge_config.patch

I didn't know this was needed, which is a separate issue, but it looks good.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/c4e4613dbe32
Remove munging of l10n-nightly configs for beta=>release migration. r=sfraser a=release for CLOSED TREE on central DONTBUILD

Comment 21

[Jordan Lund (:jlund)]

(In reply to Justin Wood (:Callek) from comment #13)
> (In reply to Francesco Lodolo [:flod] from comment #11)
> > Not sure if it needs to be reported, but we'll have a new language (ia) in
> > Beta 59, from bug 1427688.
> 
> I added a note on releng's side to validate that this locale is indeed built
> and ready to ship.

I'm going to say this is built[0] and ready to ship[1]


[0] http://archive.mozilla.org/pub/devedition/candidates/59.0b1-candidates/build11/update/win64/ia/
[1] # from balrog release blob:
         "ia": {
          "appVersion": "59.0",
          "buildID": "20180116202029",
          "completes": [
            {
              "filesize": 48156590,
              "from": "*",
              "hashValue": "8c8435ed51f60693654848bda0902bf6c637346c74d0f17597fe223f7811c7f1074081433a598f9c8b715a935d72adc0906f4136a8d46f3495966a3b231ca178"
            }
          ],
          "displayVersion": "59.0 Beta 1",
          "platformVersion": "59.0"
        },

Was there additional testing you were going to do callek?

Comment 22

[Justin Wood (:Callek)]

Nope that was it, thanks.

Comment 23

[Firefox Bug Husbandry Bot]

Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483

Comment 24

[anhtuanho]

on bug

Comment 25

[Chris AtLee [:catlee]]

Anything left to do here?

Comment 26

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.