918055 - port b2g branching script to mozharness, with revision locking

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

Bug 917433 is an example of how we're going to have to treat merge day very carefully for gecko.git.

However, for gecko l10n, we may have to switch the branch names to also be versioned (rather than train-names).

For example, https://git.mozilla.org/?p=releases/l10n/es-ES/gecko.git;a=summary currently has a mozilla-beta branch.  This is ok for now, since there is only one branch for partners to use for their localizations.  When we add the other trains, they either need to know which train corresponds to which release, or we need to name the branches differently, e.g. v1.2, and we hit the exact same timing issues that we need to resolve in bugs like bug 917433... before merge day, l10n needs to come from https://hg.mozilla.org/releases/l10n/mozilla-aurora ; after merge day it needs to come from https://hg.mozilla.org/releases/l10n/mozilla-beta , and waiting til *after* the merge means we have an ugly non-fastforward issue in the git mirrors.

Once we switch over to mozharness for hg-git, which I'm hoping is before the next 6 week cycle, we will need to deal with this freezing-and-bumping every 6 weeks for merge day^Wweek.  I kind of want it to be a script, hence this bug.  Then again, it might be a simple enough edit that it can be a line or two in docs.

Comment 1

[Aki Sasaki (not active)]

Per our discussion with the sheriffs and the b2g team, we want to lock down all 3rd-party repos to a specific revision when we create the git branches.

The <include> statements in various manifests.  I don't see any conflicts here yet (the <default> entries for including manifests are the same), but there's the possibility for divergence.  If we're going to be locking these down, I prefer to flatten the manifests like we do with b2g_bumper.  (Except keep gecko in there, and don't change the <remote>s.)

The question here is scope.  We can:

* only lock down the manifests we build, or
* lock down all the non-included manifests (flattened).

With the manifests we don't lock down, we can

* leave them, or
* nuke them.

I'm currently leaning towards either

a) only lock down the manifests we build, and leave the un-locked-down manifests as tier 3, or
b) lock down all the manifests, and nuke the included base-____.xml files once all the others are flattened.

Fabrice, do you have an opinion here?

Comment 2

[Aki Sasaki (not active)]

Going with locking down all the non-included manifests, and deleting the others.

Comment 4

[Aki Sasaki (not active)]

Attachment: b2g_branch_repos.diff

Ready for review.

I'm running this on my laptop sans --push and the diffs look ok.  I'm both able to branch the repos and lock the 3rd party revisions, and I'm able to lock 3rd party revisions without branching for existing branches (v2.0, v1.4; I'm working on v1.3t and v1.3 now).

I have logs and b2g-manifest diffs if you'd like.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8441067 [details] [diff] [review]
b2g_branch_repos.diff

Review of attachment 8441067 [details] [diff] [review]:
-----------------------------------------------------------------

Great job! I can't believe we can obsolete the initial script now!

::: mozharness/base/script.py
@@ +734,5 @@
>                                  silent=False, log_level=INFO,
>                                  tmpfile_base_path='tmpfile',
>                                  return_type='output', save_tmpfiles=False,
> +                                throw_exception=False, fatal_exit_code=2,
> +                                success_codes=None):

2 hunks failed to apply, but it shouldn't be hard to unbitrot these.

::: scripts/merge_day/b2g_branch_repos.py
@@ +386,5 @@
> +        git = self.query_exe("git", return_type="list")
> +        new_branch = self.config['branch_name']
> +        # b2g-manifest is special
> +        if os.path.exists(dirs['abs_manifest_dir']):
> +            for cmd in (git + ["reset", "--hard"], git + ["checkout", self.config["manifest_repo_revision"]]):

maybe also add "git clean -fdx"?

@@ +400,5 @@
> +            if branch_status[0] and not branch_status[1]:
> +                # We have a local branch that doesn't exist on origin
> +                self.info("Cleaning up existing %s branch from %s." % (new_branch, name))
> +                self.run_command(
> +                    git + ['branch', '-d', new_branch],

Should it be capital D (-D) here? -d deletes merged branches only?

@@ +416,5 @@
> +        new_branch = self.config['branch_name']
> +        for name, r in repos.iteritems():
> +            revision = r["revision"]
> +            cwd = os.path.join(dirs['abs_work_dir'], name)
> +            branch_status = self.check_existing_branch(new_branch, cwd)

branch_status is a bit cryptic here. maybe unpack on the fly?

local_status, remote_status = self.check_existing_branch(new_branch, cwd)

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 8441067 [details] [diff] [review]
b2g_branch_repos.diff

http://hg.mozilla.org/build/mozharness/rev/3230c9fbf560
With review comments: http://hg.mozilla.org/build/mozharness/rev/90e793f7c2aa

Comment 7

[Aki Sasaki (not active)]

Todo:

_ update docs
_ verify (possibly when branching 1.4d)
_ remove http://hg.mozilla.org/build/braindump/file/ac27869b637b/git-related/b2g-branching/b2g_branch_repos.py
_ resolve bug

Comment 8

[Aki Sasaki (not active)]

Cypress hamachi build running to verify I didn't bork anything obvious:
http://buildbot-master85.srv.releng.scl3.mozilla.com:8001/builders/b2g_cypress_hamachi_periodic/builds/0

Comment 9

[Jordan Lund (:jlund)]

3 mozharness patches are live in production :)

Comment 10

[Aki Sasaki (not active)]

Docs: https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Create_new_B2G_git_branches

Comment 11

[Aki Sasaki (not active)]

I filed bug 1029860 for cleanup.  We're not going to create a dolphin branch.  Closing, for now.