847727 - automate beagle hg-git conversion in mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

Right now we have some scripts, a lot of config files, and a lot of docs+manual steps.

I want to get this config-driven and hands-off.
Also, I want the single git->git, single hg->hg, single hg->git, or single git->hg workflows to be a subset of this workflow.

This should handle both the gecko.git and beagle scenarios.

Let's see how realistic this is.

Comment 1

[Aki Sasaki (not active)]

Attachment: check in gecko.git hgrc+config

Comment 2

[Aki Sasaki (not active)]

Soft blocker: generic retries.
Soft blocker: GitVCS object (but my current approach will require it).

Comment 3

[Aki Sasaki (not active)]

Progress here: https://github.com/escapewindow/mozharness/compare/vcs_conversion

Comment 4

[Hal Wine [:hwine] use NI!]

Comment on attachment 721009 [details] [diff] [review]
check in gecko.git hgrc+config

Review of attachment 721009 [details] [diff] [review]:
-----------------------------------------------------------------

If these are the "as is", only comment is I'd prefer releases-gecko as the directory name to be consistent. I'd actually like to move the directory to that location in the current system to reduce the number of exceptions around it (I'd coordinate that).

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 721009 [details] [diff] [review]
check in gecko.git hgrc+config

Pushed after an 'hg mv gecko releases-gecko' and commit.

http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/rev/adda5294a660

Comment 6

[Aki Sasaki (not active)]

I've got wip in https://github.com/escapewindow/mozharness/tree/vcs_conversion that

* gets its source repo / target repo / branch list from a config file, rather than hardcodes or pre-populated .hg+.git directories
* avoids hg.m.o corruption by cloning/pulling into a clean source directory, then pulling from there into a working conversion directory.  If hg.m.o corrupts the source directory, we can blow that away and recreate without corrupting the working directory, which will save us conversion time.
* creates the work directory, populates, does the conversion

The script is currently converting m-c, which should take a few days.
I still need CVS prepending info from hwine before I'll have the right SHAs.
I'll need to add pushing logic and more error detection + verification, as well as notification.

In theory this script should be able to be fed different configs to create either beagle (gecko-dev.git) or the partner-oriented gecko.git.  I'll put that to the test later, after I get to CVS-based SHA parity.

Comment 7

[Aki Sasaki (not active)]

This is *still* converting ~10 days later, potentially because I pulled b2g18 in as well for testing purposes.  I either thought it would be an incremental increase or forgot about it (most likely both).

I'm going to look at gps' hg-git patches while this runs.

Comment 8

[Aki Sasaki (not active)]

* hwine's fork conversion is still running, but the shas appear to match so far
* latest upstream hg-git finished in less than a day, but the shas have diverged.

We're trying to figure out how to deal with this -- we may need to cherry pick gps' patches onto hal's fork, or something.

Comment 10

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #8)
> * hwine's fork conversion is still running, but the shas appear to match so
> far
> * latest upstream hg-git finished in less than a day, but the shas have
> diverged.
> 
> We're trying to figure out how to deal with this -- we may need to cherry
> pick gps' patches onto hal's fork, or something.

Solved with https://github.com/escapewindow/hg-git , which is a new fork from latest upstream.

Comment 11

[Aki Sasaki (not active)]

I have cvs prepending basically working in automation, with the same shas as gecko.git.

Comment 12

[Aki Sasaki (not active)]

I now have the mapfile generated; it's fully a subset of gecko-mapfile.

Comment 13

[Aki Sasaki (not active)]

I noticed that I had made a *lot* of changes that are no longer needed (non-beagle changes, etc.) so I created a new branch and copied the needed 2 files over.

I had to remove the VCSConversionMixin import since that's no longer part of mozharness.base.vcs.vcsbase, but I wasn't using it in poc_beagle.py anyway.

https://github.com/escapewindow/mozharness/commits/beagle

Comment 14

[Aki Sasaki (not active)]

I was able to pull in mozilla-b2g18, and the mapfile is still fully a subset of gecko-mapfile.

Since I made a lot of changes since the first few actions were tested, and since we have a weekend, I made some backups and am now running a full conversion run from the beginning.

Comment 15

[Aki Sasaki (not active)]

The weekend run died at the end of prepend-cvs because it was looking for pre-cvs-mapfile at an old location (it worked on previous test runs because I hadn't clobbered the directory, so I had multiple copies of pre-cvs-mapfile lying around).

I fixed the location in the script, restarted the script from that point using the power of mozharness actions, and the script finished successfully.

The generated gecko-mapfile after pulling in m-c default and m-b2g18 default is fully a subset of git.m.o's gecko-mapfile.

Now I need to get push/upload/notify working, and the update loop error detection and recovery automated.  I'll need to be able to deal with the various not-so-edge cases.

I'm not sure how much we want to be able to bring up a new box from scratch via puppet; that's probably out of scope for this specific bug.

Comment 16

[Aki Sasaki (not active)]

The repo/branch update loop is now code complete, I think.

I now:

* update the virtualenv if new dependencies are added
* create a json file as describe in https://bugzilla.mozilla.org/show_bug.cgi?id=799845#c4
* upload to specified location(s)
* email specified people, every time or on failure only based on a flag

I need to deal with known error conditions, make this production-level bulletproof, test, and add the various other repos/branches.  I don't think I have tag support yet, and probably need to add that.

Comment 17

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #16)
> I don't think
> I have tag support yet, and probably need to add that.

Added.

Comment 18

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #17)
> (In reply to Aki Sasaki [:aki] from comment #16)
> > I don't think
> > I have tag support yet, and probably need to add that.
> 
> Added.

Sadly, while testing this, I noticed the <h<surkov email issue reared its ugly head again.  I'm going to have to investigate to see how widespread this issue is (my repo only, or also in Ehsan's github repo / Hal's conversion repo), why it's occurring, and how to either remove the issue or mitigate.

Comment 19

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #18)
> Sadly, while testing this, I noticed the <h<surkov email issue reared its
> ugly head again.  I'm going to have to investigate to see how widespread
> this issue is (my repo only, or also in Ehsan's github repo / Hal's
> conversion repo), why it's occurring, and how to either remove the issue or
> mitigate.

I think this is fixed.  Running a new conversion to test.

Comment 20

[Aki Sasaki (not active)]

I don't think this is fixed, after the previous conversion.
Re-converting to right before the tag fixing, then *backing up*, and iterating on the steps after that point.

Comment 21

[Aki Sasaki (not active)]

I've been frustrated by my inability to get rid of the <h<surkov email commit.
I think the issue is I missed porting this line from the git-filter-branch docs:

for x in `git branch | grep -v ^*`; do git filter-branch -f -- "3229d5d8b7f8376cfb7936e7be810635a14a486b..$x"; if [ -d .git-rewrite/map ]; then cp -R .git-rewrite/map ../git-rewrite/map; fi; rm -rf .git-rewrite/; done

https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

Porting that, and retrying my prepend-cvs action once I nuke my conversion dir and restore from my pre-cvs backup.

Comment 22

[(no longer active)]

(In reply to Aki Sasaki [:aki] from comment #21)
> I've been frustrated by my inability to get rid of the <h<surkov email
> commit.
> I think the issue is I missed porting this line from the git-filter-branch
> docs:
> 
> for x in `git branch | grep -v ^*`; do git filter-branch -f --
> "3229d5d8b7f8376cfb7936e7be810635a14a486b..$x"; if [ -d .git-rewrite/map ];
> then cp -R .git-rewrite/map ../git-rewrite/map; fi; rm -rf .git-rewrite/;
> done
> 
> https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

Yeah, you definitely do need that line.  Please let me know if there's something that I can do to help, most of this bug is sort of incomprehensible to my naked eyes.  :-)

Comment 23

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #22)
> (In reply to Aki Sasaki [:aki] from comment #21)
> Yeah, you definitely do need that line.

Cool, good to know.  Hopefully after this finishes, I'll get rid of that <h<surkov once and for all.

> Please let me know if there's
> something that I can do to help, most of this bug is sort of
> incomprehensible to my naked eyes.  :-)

I CCed you mainly for informational purposes, since I *think* I un-stuck myself in comment 21.

Comment 24

[Aki Sasaki (not active)]

Latest error:

Halting on failure while running ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites', '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH']

I'm going to investigate later today.

Comment 25

[Aki Sasaki (not active)]

Hm.

https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

"This repository has been disabled.

Access to this repository has been disabled by GitHub staff due to excessive use of resources. Contact support to restore access to this repository. Read here to learn more about decreasing the size of your repository. "

Comment 26

[(no longer active)]

(In reply to comment #25)
> Hm.
> 
> https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion
> 
> "This repository has been disabled.
> 
> Access to this repository has been disabled by GitHub staff due to excessive
> use of resources. Contact support to restore access to this repository. Read
> here to learn more about decreasing the size of your repository. "

Hm, yeah I received an email about this on Saturday, and have asked the github support on what I can do but they haven't gotten back to me yet.  If you have suggestions on where else I can put it please let me know.  I have a slightly out of date bundle here <http://people.mozilla.org/~eakhgari/mozilla-history-tools.bundle> in case you just want to take a look at the instructions again.

Comment 27

[(no longer active)]

(In reply to comment #26)
> (In reply to comment #25)
> > Hm.
> > 
> > https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion
> > 
> > "This repository has been disabled.
> > 
> > Access to this repository has been disabled by GitHub staff due to excessive
> > use of resources. Contact support to restore access to this repository. Read
> > here to learn more about decreasing the size of your repository. "
> 
> Hm, yeah I received an email about this on Saturday, and have asked the github
> support on what I can do but they haven't gotten back to me yet.  If you have
> suggestions on where else I can put it please let me know.  I have a slightly
> out of date bundle here
> <http://people.mozilla.org/~eakhgari/mozilla-history-tools.bundle> in case you
> just want to take a look at the instructions again.

(and filed bug 878964 to see if I can host this on git.mozilla.org/)

Comment 28

[Aki Sasaki (not active)]

Aha,
20:25:22     INFO - Running command: ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites', '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH'] in /home/asasaki/moz2/build/conversion/beagle/.git
20:25:22     INFO - Copy/paste: /home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites -- 3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH
20:25:22     INFO - Using partial env: {'PATH': '%(PATH)s:/usr/libexec/git-core'}
20:25:22     INFO -  Cannot create a new backup.
20:25:22     INFO -  A previous backup already exists in refs/original/
20:25:22     INFO -  Force overwriting the backup with -f
20:25:22    ERROR - Return code: 1

I'm going to guess that relbranch was potentially already filter-branched via another branch or something.

I can either explicitly ignore this case or use -f.
Or blacklist this branch specifically from the git-filter-branch step.

Comment 29

[(no longer active)]

(In reply to comment #28)
> Aha,
> 20:25:22     INFO - Running command:
> ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites',
> '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH']
> in /home/asasaki/moz2/build/conversion/beagle/.git
> 20:25:22     INFO - Copy/paste:
> /home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites --
> 3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH
> 20:25:22     INFO - Using partial env: {'PATH':
> '%(PATH)s:/usr/libexec/git-core'}
> 20:25:22     INFO -  Cannot create a new backup.
> 20:25:22     INFO -  A previous backup already exists in refs/original/
> 20:25:22     INFO -  Force overwriting the backup with -f
> 20:25:22    ERROR - Return code: 1
> 
> I'm going to guess that relbranch was potentially already filter-branched via
> another branch or something.
> 
> I can either explicitly ignore this case or use -f.
> Or blacklist this branch specifically from the git-filter-branch step.

Hmm, I'm not sure if I remember hitting something similar to this when I was doing the conversion...

Comment 30

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #29)
> (In reply to comment #28)
> > I'm going to guess that relbranch was potentially already filter-branched via
> > another branch or something.
> > 
> > I can either explicitly ignore this case or use -f.
> > Or blacklist this branch specifically from the git-filter-branch step.
> 
> Hmm, I'm not sure if I remember hitting something similar to this when I was
> doing the conversion...

I'm explicitly checking the status of each git-filter-branch-keep-rewrites call.
Your for loop in comment 21 could have hit this, and not errored out.  Unless you looked at the logs, you might have missed this.

It's also possible that I'm hitting something you hadn't, which I don't like the sound of.

Comment 31

[(no longer active)]

(In reply to comment #30)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #29)
> > (In reply to comment #28)
> > > I'm going to guess that relbranch was potentially already filter-branched via
> > > another branch or something.
> > > 
> > > I can either explicitly ignore this case or use -f.
> > > Or blacklist this branch specifically from the git-filter-branch step.
> > 
> > Hmm, I'm not sure if I remember hitting something similar to this when I was
> > doing the conversion...
> 
> I'm explicitly checking the status of each git-filter-branch-keep-rewrites
> call.
> Your for loop in comment 21 could have hit this, and not errored out.  Unless
> you looked at the logs, you might have missed this.

Even though I don't have that explicit check, I'm fairly certain that I watched over the output logs pretty carefully.  But it's hard to say for sure given how long has passed since then...

> It's also possible that I'm hitting something you hadn't, which I don't like
> the sound of.

Are you running the initial conversion on the same mozilla-central changeset or a more recent one?

Comment 32

[Aki Sasaki (not active)]

I'm running it on tip.
I'd be surprised if COMM193a4_20100510_RELBRANCH was recent, though.

Comment 33

[Aki Sasaki (not active)]

COMM193a4_20100510_RELBRANCH isn't in https://github.com/mozilla/mozilla-central/branches , so something is different.

Comment 34

[(no longer active)]

(In reply to comment #33)
> COMM193a4_20100510_RELBRANCH isn't in
> https://github.com/mozilla/mozilla-central/branches , so something is
> different.

Are you sure?

$ git branch -a | grep COMM193a4
  remotes/mc/COMM193a4_20100510_RELBRANCH
  remotes/origin/COMM193a4_20100510_RELBRANCH

Comment 35

[Aki Sasaki (not active)]

Ok. I still don't see it in the html page though.

Comment 36

[Aki Sasaki (not active)]

Sorry, buried in the GECKO branches. N/m me.

Comment 37

[(no longer active)]

(In reply to comment #36)
> Sorry, buried in the GECKO branches. N/m me.

Yeah... I don't find github's UI particularly useful for navigating branches...  (and tags for that matter!)  You're definitely not the first person confused by it.  :-)

Comment 38

[Aki Sasaki (not active)]

Tl;dr status update: I'm still working on this.


I'm still trying to verify that git-filter-branch-keep-rewrites will work with -f.

I've hit the following:
* prepended cvs to a restored backup of the conversion dir that only had the master branch
* blew that away, started from scratch, backed up the conversion dir before prepending cvs
* hit the above -f issue
* restored the conversion dir, restarted cvs prepend, forgot to git pull my latest changes (-f), so hit the above -f issue again
* restored the conversion dir, restarted cvs prepend, died on the mc-git-rewrite dir existing when trying to move the .git-rewrite dir out.
** added an rmtree, and fixed the move() method
* tried running the git-filter-branch-keep-rewrites -f manually, but this was taking a while
** tried running the automated cvs prepend simultaneously, ran dangerously close to 100% inode usage
** killed manual git-filter-branch-keep-rewrites, let automated cvs prepend run

I'm currently here.  I can investigate other things in parallel once bug 877922 is fixed; until then I'm stuck with single-threaded testing.

Hoping this works, so I can then a) backup the conversion dir and mc-git-rewrite dir again, b) fix tags, and c) verify that <h<surkov is gone.

Comment 39

[Aki Sasaki (not active)]

The git-filter-branch bit finished with no errors! -f worked.
I need to fix tags, then verify <h<surkov is gone.  I've already backed up the dirs so I can get back to this point without re-converting, assuming I haven't done anything wrong.

Will get back to this around buildduty emergencies.

Comment 40

[Aki Sasaki (not active)]

Good news: I was able to push to all my test repos without issue, which tells me <h<surkov may not be an issue anymore!

Bad news: my tip revision is unknown in gecko.git, meaning the shas have diverged somewhere :( :(

Tracking this down.  I currently suspect the git-filter-branch-keep-rewrites -f of an old m-c branch changed a parent revision of master somewhere.

Comment 41

[(no longer active)]

Does gecko.git have the CVS history?

Comment 42

[Aki Sasaki (not active)]

Yes, it does.  My conversion was fully a subset of gecko.git's shas until I started adding the other mozilla-central branches.

Comment 43

[(no longer active)]

(In reply to comment #42)
> Yes, it does.  My conversion was fully a subset of gecko.git's shas until I
> started adding the other mozilla-central branches.

If you take one of those branches and do a git merge-base master <branch-name>, what do you get?  That's the best way to find out where histories deviate.

Comment 44

[Aki Sasaki (not active)]

Hm,
git merge-base master COMM193a4_20100510_RELBRANCH
2249d7ec3a12c82bf01e787fffb8e7b12f9cc9a6

I'm currently suspecting that the -f wasn't the right approach, and I should be ignoring errors.  I'm going to try that approach while also trying to debug the sha divergence in a backup directory.

Comment 45

[Aki Sasaki (not active)]

I re-ran conversion without the -f.

1. Two things:

a) It's a lot faster, but the <h<surkov email is back if I run git-filter-branch-keep-rewrites without -f.
b) The shas still diverge, which means the -f wasn't the [sole] culprit in the shas diverging.


2. Potential reasons the shas might have diverged:

a) When I blew away and recreated everything, it's possible one of the python packages changed versions.  I should investigate and lock down these versions.

b) It's possible that since I'm converting a larger number of commits than I was previously, a newer commit reacted differently to an initial conversion step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it actively munges emails, etc.)

c) It's possible that the act of initially converting other m-c branches other than default by itself results in different shas than we have in gecko.git, which only has m-c default + various b2g18 branches.  If this is true we are probably in a bad spot.

d) Something else I haven't thought of.


3. Things I can try:

a) I forgot to compare the pre-cvs shas against Hal's pre-cvs releases-mozilla-central git repo during the past two conversions.  If these differ, then we've isolated the sha divergence to pre-git-filter-branch; if not, we've ruled that out.  I'm untarring my backup and doing that.

b) Hal walked me through finding sha divergence earlier (after comment 8) but I've forgotten how.  I think I may need help here again.


Needinfo?ing for 3b.

Comment 46

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #45) 
> 3. Things I can try:
> 
> a) I forgot to compare the pre-cvs shas against Hal's pre-cvs
> releases-mozilla-central git repo during the past two conversions.  If these
> differ, then we've isolated the sha divergence to pre-git-filter-branch; if
> not, we've ruled that out.  I'm untarring my backup and doing that.

I just did a |git log| inside my restored pre-cvs backup and verified that the newest sha existed inside Hal's pre-cvs releases-mozilla-central git repo.

The restored pre-cvs backup also has an expected sha of 2514a423aca5d1273a842918589e44038d046a51 for the "Free the (distributed) Lizard!" commit.

The sha divergence comes during or after the cvs-prepend step.

Comment 47

[(no longer active)]

(In reply to comment #45)
> a) It's a lot faster, but the <h<surkov email is back if I run
> git-filter-branch-keep-rewrites without -f.
> b) The shas still diverge, which means the -f wasn't the [sole] culprit in the
> shas diverging.

:(

> 2. Potential reasons the shas might have diverged:
> 
> a) When I blew away and recreated everything, it's possible one of the python
> packages changed versions.  I should investigate and lock down these versions.

Which python packages?

> b) It's possible that since I'm converting a larger number of commits than I
> was previously, a newer commit reacted differently to an initial conversion
> step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> actively munges emails, etc.)

The only way that would be possible is if hg-git was modified after the previous initial conversion, which it was.  The only thing that can affect the sha1 of a commit based on the parent(s) is the sha1 for that parent commit.

> c) It's possible that the act of initially converting other m-c branches other
> than default by itself results in different shas than we have in gecko.git,
> which only has m-c default + various b2g18 branches.  If this is true we are
> probably in a bad spot.

I don't see how this could possibly be the case.

> d) Something else I haven't thought of.
> 
> 3. Things I can try:
> 
> a) I forgot to compare the pre-cvs shas against Hal's pre-cvs
> releases-mozilla-central git repo during the past two conversions.  If these
> differ, then we've isolated the sha divergence to pre-git-filter-branch; if
> not, we've ruled that out.  I'm untarring my backup and doing that.
> 
> b) Hal walked me through finding sha divergence earlier (after comment 8) but
> I've forgotten how.  I think I may need help here again.

git merge-base?  Or are you asking how to use merge-base?

Comment 48

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #47)
> (In reply to comment #45)
> > 2. Potential reasons the shas might have diverged:
> > 
> > a) When I blew away and recreated everything, it's possible one of the python
> > packages changed versions.  I should investigate and lock down these versions.
> 
> Which python packages?

01:00:38     INFO - Current package versions:
01:00:38     INFO -   mapper == 0.1
01:00:38     INFO -   ordereddict == 1.1
01:00:38     INFO -   dulwich == 0.9.0
01:00:38     INFO -   bottle == 0.11.6
01:00:38     INFO -   distribute == 0.6.28
01:00:38     INFO -   hg-git == 0.4.0-moz2
01:00:38     INFO -   mercurial == 2.2.1

> > b) It's possible that since I'm converting a larger number of commits than I
> > was previously, a newer commit reacted differently to an initial conversion
> > step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> > actively munges emails, etc.)
> 
> The only way that would be possible is if hg-git was modified after the
> previous initial conversion, which it was.  The only thing that can affect
> the sha1 of a commit based on the parent(s) is the sha1 for that parent
> commit.

I think the sha differs if you change <h<surkov to <h?surkov or <hsurkov , no?

Basically, git-filter-branch changes things in the commits, like timestamps and email addresses.  You ran git-filter-branch against N number of commits, Hal ran it later against N + N1 number of commits, I ran it later against N + N1 + N2 number of commits.  This time I ran it against N + N1 + N2 + N3 number of commits, and I'm hypothesizing that git-filter-branch munged something in that last N3 batch of commits to make my shas no longer match.

> > c) It's possible that the act of initially converting other m-c branches other
> > than default by itself results in different shas than we have in gecko.git,
> > which only has m-c default + various b2g18 branches.  If this is true we are
> > probably in a bad spot.
> 
> I don't see how this could possibly be the case.

Ok.

> > b) Hal walked me through finding sha divergence earlier (after comment 8) but
> > I've forgotten how.  I think I may need help here again.
> 
> git merge-base?  Or are you asking how to use merge-base?

We're hypothesizing you mean a) pull gecko.git into my working conversion dir and b) try to see the common ancestor between the two master revisions?

The method Hal showed me before is

git --git-dir build/conversion/beagle/.git log --format="%H %P %ai %s" --reverse --topo-order > shas

And do the same thing with gecko.git, then diff the sha text files.  The first line that diffs should be the first divergent sha.

I'll try this method first, since it's less destructive to my working dir, and I'm already short on disk+inodes.

Comment 49

[(no longer active)]

(In reply to comment #48)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #47)
> > (In reply to comment #45)
> > > 2. Potential reasons the shas might have diverged:
> > > 
> > > a) When I blew away and recreated everything, it's possible one of the python
> > > packages changed versions.  I should investigate and lock down these versions.
> > 
> > Which python packages?
> 
> 01:00:38     INFO - Current package versions:
> 01:00:38     INFO -   mapper == 0.1
> 01:00:38     INFO -   ordereddict == 1.1
> 01:00:38     INFO -   dulwich == 0.9.0
> 01:00:38     INFO -   bottle == 0.11.6
> 01:00:38     INFO -   distribute == 0.6.28
> 01:00:38     INFO -   hg-git == 0.4.0-moz2
> 01:00:38     INFO -   mercurial == 2.2.1

I think only hg-git matters in this list.

> > > b) It's possible that since I'm converting a larger number of commits than I
> > > was previously, a newer commit reacted differently to an initial conversion
> > > step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> > > actively munges emails, etc.)
> > 
> > The only way that would be possible is if hg-git was modified after the
> > previous initial conversion, which it was.  The only thing that can affect
> > the sha1 of a commit based on the parent(s) is the sha1 for that parent
> > commit.
> 
> I think the sha differs if you change <h<surkov to <h?surkov or <hsurkov , no?

Yes.  git commits are basically plaintext files of the following format:

tree f7722149a10a1347075520160b3cbd1ef4ef6a40
parent 062746d6f8015f68f72a593bdee720955f71ffcf
author Ehsan Akhgari <[e-mail]> 1371518325 -0400
committer Ehsan Akhgari <[e-mail]> 1371681249 -0400

Commit message

The commit sha1 is the sha1 of this text file.  What I meant to say is that the only thing from the parent commit that affects the sha1 of this commit is the sha1 of the parent commit.  So if uyyou do anything to change the sha1 of 1 commit, everything will diverge from that point on.

> Basically, git-filter-branch changes things in the commits, like timestamps and
> email addresses.  You ran git-filter-branch against N number of commits, Hal
> ran it later against N + N1 number of commits, I ran it later against N + N1 +
> N2 number of commits.  This time I ran it against N + N1 + N2 + N3 number of
> commits, and I'm hypothesizing that git-filter-branch munged something in that
> last N3 batch of commits to make my shas no longer match.

If the merge base is within N3, then you're right.

> > > b) Hal walked me through finding sha divergence earlier (after comment 8) but
> > > I've forgotten how.  I think I may need help here again.
> > 
> > git merge-base?  Or are you asking how to use merge-base?
> 
> We're hypothesizing you mean a) pull gecko.git into my working conversion dir
> and b) try to see the common ancestor between the two master revisions?

Yes.

> The method Hal showed me before is
> 
> git --git-dir build/conversion/beagle/.git log --format="%H %P %ai %s"
> --reverse --topo-order > shas
> 
> And do the same thing with gecko.git, then diff the sha text files.  The first
> line that diffs should be the first divergent sha.

I believe this should result in the same thing, it will only be slower...  But I haven't tested this.  git-merge-base is how git finds the merge base for any two commits, and is going to be way less error prone.

Comment 50

[Aki Sasaki (not active)]

May 19.

< 8962237ccb5c517d8a63ac681c89e758afda3bdb d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test
> 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test

From gecko.git:

commit 8962237ccb5c517d8a63ac681c89e758afda3bdb
Author: Carlos G. <[e-mail]>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test


From my conversion dir:

commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646
Author: Carlos G <[e-mail]>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test



Comment 15, where my conversion dir was fully a subset of gecko.git, was May 13.
My theory is that git-filter-branch-keep-rewrites is stripping the trailing '.' from "Carlos G."

I may have to limit the initial conversion to stop before this revision, git-filter-branch-keep-rewrites with -f, and then convert the rest with just hg-git.

Not entirely sure how to find the parent revision's hg revision, but I'll figure it out.

Comment 51

[Aki Sasaki (not active)]

Hm, I should verify the new version of hg-git (0.4.0-moz2) isn't munging this before I go through the whole |git-filter-branch-keep-rewrites -f| multi-day conversion.

Comment 52

[(no longer active)]

(In reply to Aki Sasaki [:aki] from comment #50)
> May 19.
> 
> < 8962237ccb5c517d8a63ac681c89e758afda3bdb
> d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug
> 463696 - crash test
> > 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test
> 
> From gecko.git:
> 
> commit 8962237ccb5c517d8a63ac681c89e758afda3bdb
> Author: Carlos G. <[e-mail]>
> Date:   Sun May 19 18:20:20 2013 +0000
> 
>     Bug 463696 - crash test
> 
> 
> From my conversion dir:
> 
> commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646
> Author: Carlos G <[e-mail]>
> Date:   Sun May 19 18:20:20 2013 +0000
> 
>     Bug 463696 - crash test
> 
> 
> 
> Comment 15, where my conversion dir was fully a subset of gecko.git, was May
> 13.
> My theory is that git-filter-branch-keep-rewrites is stripping the trailing
> '.' from "Carlos G."

That seems like a bug. :(

> I may have to limit the initial conversion to stop before this revision,
> git-filter-branch-keep-rewrites with -f, and then convert the rest with just
> hg-git.
> 
> Not entirely sure how to find the parent revision's hg revision, but I'll
> figure it out.

Just look in the git-mapfile.

Comment 53

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #51)
> Hm, I should verify the new version of hg-git (0.4.0-moz2) isn't munging
> this before I go through the whole |git-filter-branch-keep-rewrites -f|
> multi-day conversion.

In my pre-cvs-prepend backup, the '.' is kept:

commit 7bcdb26c4e194e80744f4beb76f2df58b4bb475e
Author: Carlos G. <[e-mail]>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test

That points the finger directly at the cvs-prepend steps, primarily git-filter-branch-keep-rewrites.

Comment 54

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #52)
> (In reply to Aki Sasaki [:aki] from comment #50)
> > I may have to limit the initial conversion to stop before this revision,
> > git-filter-branch-keep-rewrites with -f, and then convert the rest with just
> > hg-git.
> > 
> > Not entirely sure how to find the parent revision's hg revision, but I'll
> > figure it out.
> 
> Just look in the git-mapfile.

This is the problem commit:
http://hg.mozilla.org/mozilla-central/rev/82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2

The theory is, if I limit the clone for the initial conversion to stop at its parent ( http://hg.mozilla.org/mozilla-central/rev/2701855384d6 ) then git-filter-branch won't munge this commit, and subsequent hg-git passes will do the right thing, and shas won't diverge.

Comment 55

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #54)
> This is the problem commit:
> http://hg.mozilla.org/mozilla-central/rev/
> 82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2
> 
> The theory is, if I limit the clone for the initial conversion to stop at
> its parent ( http://hg.mozilla.org/mozilla-central/rev/2701855384d6 ) then
> git-filter-branch won't munge this commit, and subsequent hg-git passes will
> do the right thing, and shas won't diverge.

I think I can clone, then use hg strip:
http://mercurial.selenic.com/wiki/PruningDeadBranches#Using_strip

hg --config extensions.mq= strip -n 82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2

Comment 56

[Aki Sasaki (not active)]

Verified that the same commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 exists in the |git-filter-branch-keep-rewrites -f| backup, and that the final tip-of-master sha exists in my current working directory.

I think if I do the above |hg strip|, and run git-filter-branch-keep-rewrites with -f, I will fix both the sha divergence and the <h<surkov email problem.

I'm going to start that today; judging by the last attempt, this may take til mid-next week.

Comment 57

[Aki Sasaki (not active)]

Converted, prepended cvs, fixed tags.
As of right now, the tip of master's sha exists in gecko.git (non-divergent shas for master branch).

I'm going to back up the working dir and then do a conversion loop get the rest of may 16 -> today converted + pushed.  (The push is also where we detect the absence or existence of the <h<surkov email.)

Comment 58

[Aki Sasaki (not active)]

...aaaaaaand we have both divergent shas and <h<surkov.

My current hypothesis is the hg strip left behind some unstripped revisions that weren't "tip of master", but children of tip-of-master.  So they weren't git-filter-branched and screwed things up.  If this is the case, nuking them from orbit before the push loop (or maybe before the tag fixing, or maybe before the cvs prepending, depending) may fix things.

I will debug in a bit.

Comment 59

[Aki Sasaki (not active)]

<h<surkov:

* The offending revision is b294eb35d142755f7724ed5bf96f11d2d4054a1b
* |git branch --contains b294eb35d142755f7724ed5bf96f11d2d4054a1b| only gives "master"
** I had already started to wonder if my |hg strip| was faulty, since "default" had a number of children.  I'm guessing I need to strip further up so tip-of-default has zero children before I start the initial conversion.

Diverging shas:
* I have two separate "Free the (distributed) Lizard!" commits in my conversion dir.
** One is 05e5d33a570d48aed58b2d38f5dfc0a7870ff8d3 , which matches gecko.git
*** |git branch --contains 05e5d33a570d48aed58b2d38f5dfc0a7870ff8d3| gives me a lot of (all?) branches, including a "* master".
*** This revision shows in my post-cvs-mapfile and gecko-mapfile, but not the pre-cvs-mapfile
** The other is 2514a423aca5d1273a842918589e44038d046a51 , which doesn't exist in gecko.git
*** |git branch --contains 2514a423aca5d1273a842918589e44038d046a51| gives "master"
*** This revision shows in my pre-cvs-mapfile, but not the post-cvs-mapfile or gecko-mapfile

I traced http://hg.mozilla.org/mozilla-central/rev/82e4f1b7bbb6 's children until the merge -- I think the issue is the graph doesn't allow for cleanly stripping that single revision (maybe?): http://hg.mozilla.org/mozilla-central/graph/132363

I think keeping the same approach, but stripping all revisions that came after http://hg.mozilla.org/mozilla-central/rev/317fe0f314ab might be one approach.  Verifying this via re-converting will be time consuming, so I want to do some more digging first, but I currently don't see anything glaring to prove this line of thinking wrong.

Comment 60

[Aki Sasaki (not active)]

After I hg strip these revisions: ("26cb30a532a1", "aad29aa89237", "eb1d7f3cd1d7", "9f2fa4839e98"), revision 317fe0f314ab shows up as 'tip', and it appears to have no child revisions:

deathduck:/src/vcs_conversion/hg-mc [15:52:44] (default)
534$ hg log -r 132332
changeset:   132332:317fe0f314ab
tag:         tip
parent:      132331:1196b497640b
parent:      132290:8ca260fe91e3
user:        Phil Ringnalda <[e-mail]>
date:        Sat May 18 18:07:46 2013 -0700
summary:     Merge the last PGO-green cset from m-i to m-c

deathduck:/src/vcs_conversion/hg-mc [15:52:48] (default)
535$ hg log -r 132333
abort: unknown revision '132333'!

An |hg glog| appears to confirm.  I'm going to pull backups of this conversion run to gd3 for later investigation, if desired, then I'm going to start another conversion run with the new hg strip logic.

Comment 61

[Aki Sasaki (not active)]

Status:

I started off a conversion run after comment 60.  The initial conversion went smoothly; I backed up the conversion directory afterwards, and verified that the tip-of-master sha existed in the releases-mozilla-central github repo (this repo doesn't have CVS history, so the shas should match).  Good.

Afterwards, I kicked off the cvs prepend step, and crossed my fingers, since this would [dis?]prove my hg strip theory.  And I let it run, as the variopus git-filter-branch-keep-rewrites steps for all the m-c branches combine for around a week of compute time.

On Wednesday, the machine rebooted to resolve bug 894225.  I lost nothing that I couldn't reproduce, but I did lose 4 days of prepend-cvs compute time.  I didn't have any way of restarting the prepend-cvs step midway, and didn't trust that either.  I restored my initial conversion backup, wrote a patch to run incremental backups of the conversion dir after each branch, then restarted the prepend-cvs step yesterday.

With any luck, this will finish late next week and we'll be able to verify whether the hg strip fixes result in a good conversion.

Comment 62

[Aki Sasaki (not active)]

...... I think we're good!!

This finished, and pushed to github without issue (this is how I caught the <h<surkov issue before).

The latest changeset on master, 9a1a3a0658790de7f95e0ab4a18ba78f71266b96, exists on gecko.git:
http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=9a1a3a0658790de7f95e0ab4a18ba78f71266b96

I only pushed the master/b2g18{,_v_1_0_0,_v_1_0_1} branches to https://github.com/escapewindow/test-beagle , but it's there for people to look at.

I still need to:

* add the rest of the branches that we want to be a part of this repo
* relatively minor code cleanup
* split the patch up into ~5 patches for easier review

for the script portion.  Automating running this script should be relatively easy, as well as questions like bug 847640.

Comment 63

[Aki Sasaki (not active)]

While populating some of the rest of the branches, I noticed that a number of the FIREFOX_/FENNEC_ etc. tags aren't getting populated.
https://github.com/escapewindow/test-beagle/branches

If I were only supporting beagle, I could potentially pull in all branches and tags, and allow for moving tags (releng release automation regularly moves {FIREFOX,FENNEC}_*_RELEASE tags; git doesn't like tag moves).  However, I've been keeping an eye out for supporting the partner-oriented gecko.git as well, in which case moving tags are verboten.

I probably need to change my repo/branch/tag definitions:
https://github.com/escapewindow/mozharness/blob/beagle/configs/users/aki/beagle1.py#L50

* branches and tags need:
** incoming config (superset of all branches+tags needed), with whitelist, blacklist?, regexes, and potential renames.
** outgoing config, which can be global or per-target, with whitelist, blacklist?, regexes, and potential renames.
* ideally some sort of config verification, so we don't make a config change to the production process that pushes bad changes to public repos
* possibly make it easier to deal with than creating one massive dict

The feedback loop in testing this should be massively faster than the cvs prepend feedback loop, though, so this should be a relatively fast change to make.


Also, I'm going to write a blog post about my work here in the next couple days.

Comment 64

[Aki Sasaki (not active)]

http://escapewindow.dreamwidth.org/238476.html

Comment 65

[Aki Sasaki (not active)]

Merged my beagle_branches branch into the beagle branch, then got this:



02:48:48    ERROR -  abort: Parent SHA-1 not present in Gitrepo: 774108fb3240461699cd5390a08bfe8c75f8768d
02:48:48    ERROR -  Automation Error: hg not responding
02:48:49    ERROR - Return code: 65280
02:49:04    ERROR -  error: src refspec refs/heads/GECKO140_2012061213_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE80_2011102618_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE70_2011083009_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE150_2012071710_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO90_2011110909_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO150_2012082116_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO150_2012082118_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO100_2012011108_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE90_2011121217_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO190_2013010816_RELBRANCH does not match any.
02:49:04    ERROR - Return code: 256
02:49:04    ERROR - mozilla-beta: Can't push /home/asasaki/beagle1/build/conversion/beagle to /home/asasaki/beagle1/build/target/beagle/.git!
<snip>

Looks like I'll need a real fix for bug 908369, not just my overwrite workaround.

Comment 66

[Aki Sasaki (not active)]

Found it, I think.

lrwxrwxrwx 1 asasaki asasaki       107 Jul 18 01:44 pack-0f19f543bfed791cc99f3168547e28cd86598c3e.idx -> /home/asasaki/moz2/build/mozilla-cvs-history/objects/pack/pack-0f19f543bfed791cc99f3168547e28cd86598c3e.idx
lrwxrwxrwx 1 asasaki asasaki       108 Jul 18 01:44 pack-0f19f543bfed791cc99f3168547e28cd86598c3e.pack -> /home/asasaki/moz2/build/mozilla-cvs-history/objects/pack/pack-0f19f543bfed791cc99f3168547e28cd86598c3e.pack

On gd2, I was in /home/asasaki/moz2/.  On gd3, I'm in /home/asasaki/beagle1/.
When I rsynced over the conversion dir, these absolute path softlinks were pointing to nothing.

When I softlinked beagle1 to moz2, this started working again.  Verifying after this next conversion pass.

Comment 67

[Aki Sasaki (not active)]

Attachment: beagle.diff

56kb, and months of changes.
Let me know if you have any questions; thanks in advance :)

Comment 68

[Rail Aliiev [:rail]]

Comment on attachment 796368 [details] [diff] [review]
beagle.diff

Review of attachment 796368 [details] [diff] [review]:
-----------------------------------------------------------------

I didn't test it (!), but in overall it looks good.

See some minor suggestions below.

::: configs/vcs_sync/beagle.py
@@ +13,5 @@
> +config = {
> +    "log_name": "beagle",
> +    "log_max_rotate": 99,
> +    "repos": [{
> +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",

Should we move these repos to /build?

@@ +43,5 @@
> +                "default": "master",
> +            },
> +        },
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,

the final production version should use something different

@@ +46,5 @@
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> +    "cvs_manifest": CVS_MANIFEST,
> +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",

The same here. Maybe something that could be publicly accessible?

::: scripts/hg_git.py
@@ +938,5 @@
> +        for notify_config in c.get('notify_config', []):
> +            if not fatal and notify_config.get('failure_only'):
> +                continue
> +            fromaddr = notify_config.get('from', c['default_notify_from'])
> +            message = string.join((

string is deprecated, use "\r\n".join(<...>) instead?

Comment 69

[Aki Sasaki (not active)]

Comment on attachment 796368 [details] [diff] [review]
beagle.diff

https://hg.mozilla.org/build/mozharness/rev/de02a1b7786d

Woot!

I did fix the string -> '\r\n'.join().  The user-specific stuff is in there for test-beagle; we should fix when we go to real beagle.

Comment 70

[Aki Sasaki (not active)]

Attachment: base.diff

part 1 of ?

This patch:

* gets rid of some print's in mozharness.base.config
* updates GitErrorList
* fixes a virtualenv log line
* removes a duplicate copy/paste log line from get_output_from_command()
* allows add_failure() to not change the exit status (return_code)
* some whitespace tweaks

Comment 71

[Aki Sasaki (not active)]

Attachment: script_cleanup

Not a required patch, just cleanup I did while I was in here.

* chmod 755 scripts/*.py
* removes obsolete device_talosrunner.py (obsoleted by android_panda_talos.py) and unfinished/old mozmill_updates.py

Comment 72

[Aki Sasaki (not active)]

Attachment: vcs_sync

Sorry about this.

Rail for code review, hwine for feedback about configs, if they're understandable without grokking the code.

This patch is currently staging-oriented, but the production changes should be minimal.
Let me know if there's anything I can do to make this review easier.

* renames the github ssh key since it's currently an additional throwaway key that I added to my github account.
* points at the new local pypi servers
* adds gecko-git, l10n, and project-branches configs.  The l10n config is skippable for this commit if there are concerns.
* creates a shared VCSSyncScript object that currently only deals with notification.  I wanted to avoid touching the initial_beagle.py script too much, since testing changes requires a week of compute time.
* splits hg_git.py into vcs-sync/initial_beagle.py and vcs-sync/vcs_sync.py so changes to the evolving vcs_sync.py don't put initial_beagle.py at too much risk.
* initial_beagle
** fixes a stupid bug that made us skip any branches left over (if the branches lenght wasn't divisible by 10)
** gets rid of the update-/push/upload steps and helper methods from initial_beagle, since we don't need them.
** hg strip's another errant head that caused <h<surkov to appear again; adds a check to fatal() if 317fe0f314ab is not the only m-c head so we don't waste more time in the future
* vcs_sync.py
** gets rid of the pull/create-/initial-conversion/prepend-cvs/fix-tags steps, since those aren't needed in the update loop
** adds support for multiple conversion dirs (beagle+gecko.git all happen in one; build/ and l10n repos want their own conversion dirs)
** adds support for defining locales and project branches by name, and generates the conversion repo dicts for you (to make it easier for us)
** makes the loop noop for a repo if there are no incoming changes.  This cuts our noop time from ~550 seconds to ~20 seconds (dependent on hg.m.o load)
** adds mapfile combining support, but doesn't use it yet
** adds first-time support to update_work_mirror() since the create_work_mirror() method was removed
** adds more HgErrorList checking
** runs gexport per repo, rather than one time for the entire conversion dir (needed for multi-conversion-dirs)
** allows for mapfile renaming per repo

Comment 73

[Aki Sasaki (not active)]

Comment on attachment 815544 [details] [diff] [review]
base.diff

http://hg.mozilla.org/build/mozharness/rev/67d0c83ee405

Comment 74

[Hal Wine [:hwine] use NI!]

Comment on attachment 815574 [details] [diff] [review]
vcs_sync

Review of attachment 815574 [details] [diff] [review]:
-----------------------------------------------------------------

gecko-git.py matches what the legacy config shows

Everything that I can easily check looks okay.

I'd love to see some additional documentation on the configs -- whether that's like a sample config with comments (ala Apache) or lighter weight annotations in each config file.

Also, for the future, it'd be great to have a config sanity checker -- it sounds like there are some "shouldn't go together", and shadowed configs that could lead to confusion years from now.

::: configs/vcs_sync/gecko-git.py
@@ +1,4 @@
> +import os
> +import socket
> +hostname = socket.gethostname()
> +

I don't see any mapfile_name specified here, but we need a mapfile. Where is that handled?

@@ +13,5 @@
> +config = {
> +    "log_name": "gecko-git",
> +    "log_max_rotate": 99,
> +    "repos": [{
> +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",

Curious as to what is being used from this repo. It may make sense to get it somewhere more permanent if things are going to live on.

@@ +41,5 @@
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> +    "cvs_manifest": CVS_MANIFEST,
> +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",

Shouldn't the initial artifacts be in tooltool?

@@ +96,5 @@
> +            "branches": {
> +                "default": "gecko-18",
> +            },
> +        },
> +        "tag_config": {

This tag_config is redundant, as both targets already have one specified.

Not a functional issue, but it can leave humans confused about which wins :)

::: configs/vcs_sync/l10n.py
@@ +1,5 @@
> +from copy import deepcopy
> +import os
> +import socket
> +hostname = socket.gethostname()
> +

I don't see any mapfile specifications here, but we do need one integrated mapfile for all l10n.

How is that handled?

::: configs/vcs_sync/project-branches.py
@@ +7,5 @@
> +PROJECT_BRANCHES = [
> +    # twig projects
> +    "ash",
> +    "alder",
> +    "ash",

duplicate twig "ash"

::: mozharness/base/vcs/vcssync.py
@@ +11,5 @@
> +import smtplib
> +import sys
> +import time
> +
> +sys.path.insert(1, os.path.dirname(os.path.dirname(os.path.dirname(sys.path[0]))))

Wow! that's cryptic. Is this a mozharnessism?

@@ +66,5 @@
> +                "To: %s" % notify_config['to'],
> +                "CC: %s" % ','.join(notify_config.get('cc', [])),
> +                "Subject: %s" % subject,
> +                "",
> +                text

SMTP lawyer here -- the body of the message is also supposed to use network format \r\n, but 'text' is built using only '\n'

In this day and age, it'll probably work, but :)

Comment 75

[Aki Sasaki (not active)]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #74)
> gecko-git.py matches what the legacy config shows
>
> Everything that I can easily check looks okay.

Cool, thanks!

> I'd love to see some additional documentation on the configs -- whether
> that's like a sample config with comments (ala Apache) or lighter weight
> annotations in each config file.

Sure.

> Also, for the future, it'd be great to have a config sanity checker -- it
> sounds like there are some "shouldn't go together", and shadowed configs
> that could lead to confusion years from now.

Sounds like bug 699343, which would be awesome to fix, but I don't know where to start.
And/or completely pulling the conversion_repos portion out of the config file into a separate file that the config file references.  But the current format is my best effort at this point in time for the format.  Maybe when the back end is explained more it'll make more sense.

> ::: configs/vcs_sync/gecko-git.py
> @@ +1,4 @@
> > +import os
> > +import socket
> > +hostname = socket.gethostname()
> > +
>
> I don't see any mapfile_name specified here, but we need a mapfile. Where is
> that handled?

It defaults to gecko-mapfile in the code; it's only overridden in the configs.
If we had apache-style comments we might have gecko-mapfile in the config, but commented out.

> @@ +13,5 @@
> > +config = {
> > +    "log_name": "gecko-git",
> > +    "log_max_rotate": 99,
> > +    "repos": [{
> > +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",
>
> Curious as to what is being used from this repo. It may make sense to get it
> somewhere more permanent if things are going to live on.

As mentioned in IRC, git-filter-branch-keep-rewrites.

> @@ +41,5 @@
> > +    },
> > +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> > +    "cvs_manifest": CVS_MANIFEST,
> > +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> > +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",
>
> Shouldn't the initial artifacts be in tooltool?

gd{2,3,4} can't reach tooltool, so this was deliberate.
I may change if the AWS boxes can, but then again I don't want to touch the initial conversion stuff too much now.

> @@ +96,5 @@
> > +            "branches": {
> > +                "default": "gecko-18",
> > +            },
> > +        },
> > +        "tag_config": {
>
> This tag_config is redundant, as both targets already have one specified.
>
> Not a functional issue, but it can leave humans confused about which wins :)

I can explain on a whiteboard maybe.
But basically: since the push is happening from a multi-repo conversion dir, and we don't save which tags were converted, a wildcard/regex tag_config for push can push tags from other repos (repo1 tag conversion is loose and gets uglytag1, prodtag1; repo2 tag conversion has a stricter conversion tag_config so it gets prodtag2 but skips uglytag2.  repo1 target tag push only pushes prodtag* so we get prodtag1 and prodtag2 downstream.  repo2 target pushes *, so we get uglytag1, prodtag1, prodtag2 downstream.  Because of this, I'm limiting on both ends for safety).

> ::: configs/vcs_sync/l10n.py
> @@ +1,5 @@
> > +from copy import deepcopy
> > +import os
> > +import socket
> > +hostname = socket.gethostname()
> > +
>
> I don't see any mapfile specifications here, but we do need one integrated
> mapfile for all l10n.
>
> How is that handled?

l10n is unfinished.  To be written.

> ::: configs/vcs_sync/project-branches.py
> @@ +7,5 @@
> > +PROJECT_BRANCHES = [
> > +    # twig projects
> > +    "ash",
> > +    "alder",
> > +    "ash",
>
> duplicate twig "ash"

Fixed.

> ::: mozharness/base/vcs/vcssync.py
> @@ +11,5 @@
> > +import smtplib
> > +import sys
> > +import time
> > +
> > +sys.path.insert(1, os.path.dirname(os.path.dirname(os.path.dirname(sys.path[0]))))
>
> Wow! that's cryptic. Is this a mozharnessism?

Yeah, basically adds ../../.. to the path to import python modules from.

> @@ +66,5 @@
> > +                "To: %s" % notify_config['to'],
> > +                "CC: %s" % ','.join(notify_config.get('cc', [])),
> > +                "Subject: %s" % subject,
> > +                "",
> > +                text
>
> SMTP lawyer here -- the body of the message is also supposed to use network
> format \r\n, but 'text' is built using only '\n'
>
> In this day and age, it'll probably work, but :)

It does work; I've been getting plenty of email.

Comment 76

[Aki Sasaki (not active)]

https://groups.google.com/d/msg/mozilla.dev.planning/hfZfm7vAHKA/fJNSQFlyUtIJ

Comment 77

[Rail Aliiev [:rail]]

Comment on attachment 815574 [details] [diff] [review]
vcs_sync

Review of attachment 815574 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

::: scripts/vcs-sync/vcs_sync.py
@@ +271,4 @@
>          else:
> +            self.all_repos = list(self.config.get('conversion_repos', []))
> +        if self.config.get('conversion_type') == 'project-branches':
> +            self.all_repos += self._query_project_repos()

Does this mean that we have all project branches in git as branches? My only concern here is the fact that we reset project branches from time to time. Would this break conversion?

Comment 78

[Aki Sasaki (not active)]

(In reply to Rail Aliiev [:rail] from comment #77)
> Comment on attachment 815574 [details] [diff] [review]
> vcs_sync
> 
> Review of attachment 815574 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> LGTM!
> 
> ::: scripts/vcs-sync/vcs_sync.py
> @@ +271,4 @@
> >          else:
> > +            self.all_repos = list(self.config.get('conversion_repos', []))
> > +        if self.config.get('conversion_type') == 'project-branches':
> > +            self.all_repos += self._query_project_repos()
> 
> Does this mean that we have all project branches in git as branches? My only
> concern here is the fact that we reset project branches from time to time.
> Would this break conversion?

Yes, https://github.com/mozilla/integration-gecko-projects/branches .
I tested with a user repo -- I landed a new non-fastforward head, then I reset the repo.  The non-fastforward head required manual fixing via |hg debugsetparents|.  The reset worked without human intervention, since the new head (based off m-c) was a fastforward from the old head.

If we get borked, we have the option of deleting the branch in github.  If we get completely borked, we have the option of deleting the entire integration-gecko-projects repo and starting over, as stated here: https://github.com/mozilla/integration-gecko-projects/blob/master/README.md

Comment 79

[Aki Sasaki (not active)]

Comment on attachment 815555 [details] [diff] [review]
script_cleanup

http://hg.mozilla.org/build/mozharness/rev/5b178e665fa0

Comment 80

[Aki Sasaki (not active)]

Comment on attachment 815574 [details] [diff] [review]
vcs_sync

http://hg.mozilla.org/build/mozharness/rev/cd8ea7755bb4

Comment 81

[Aki Sasaki (not active)]

Attachment: latest changes

This patch gets us updated to the code I'm currently running on vcssync{1,2}.

* adds gitmo-beagle target to beagle.py
* updates github targets to official targets, rather than my user account for beagle + project-branches
* changes the upload locations to people.m.o, for now, for beagle + project-branches
* removes duplicate "ash" from the project list
* adds release+vcs2vcs to the notify list
* adds some copy_to_upload_dir() updates.  I'm now copying logs to the upload dir twice: once before the upload, and once at the end of the job (since there are still steps that can go wrong after the first copy)
* adds a --no-check-incoming option, which kind of forces an update-all and push-all

Asking :hwine since :rail's out til Wednesday.  Let me know if you have questions?

Also, I think I'll close this bug and reopen new enhancement bugs after this lands.

Comment 82

[Justin Wood (:Callek)]

something here is in mozharness production

Comment 83

[Hal Wine [:hwine] use NI!]

Comment on attachment 816264 [details] [diff] [review]
latest changes

Review of attachment 816264 [details] [diff] [review]:
-----------------------------------------------------------------

The questions I have may have reasonable answers, so r+ assuming that is the case.

::: mozharness/base/script.py
@@ +1126,5 @@
>  
>              if not post_success:
>                  self.fatal("Aborting due to failure in post-run listener.")
> +        if self.config.get("copy_logs_post_run", True):
> +            self.copy_logs_to_upload_dir()

Just a question -- if we can't post, wouldn't we especially want the logs copied? Only true if the logs are useful for debugging that issue -- your call on how useful they would be.

::: scripts/vcs-sync/vcs_sync.py
@@ +781,5 @@
>          self._write_repo_update_json(repo_map)
>          if failure_msg:
>              self.fatal("Unable to push these repos:\n%s" % failure_msg)
>  
> +    def preflight_upload(self):

This is a new method, but I don't see it being invoked anywhere in this changeset.

If this was a previously un-implemented method on the base class, shouldn't it include a super() call for safety? If not, where is it being invoked?

Comment 84

[Aki Sasaki (not active)]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #83)
> Comment on attachment 816264 [details] [diff] [review]
> latest changes
> 
> Review of attachment 816264 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> The questions I have may have reasonable answers, so r+ assuming that is the
> case.
> 
> ::: mozharness/base/script.py
> @@ +1126,5 @@
> >  
> >              if not post_success:
> >                  self.fatal("Aborting due to failure in post-run listener.")
> > +        if self.config.get("copy_logs_post_run", True):
> > +            self.copy_logs_to_upload_dir()
> 
> Just a question -- if we can't post, wouldn't we especially want the logs
> copied? Only true if the logs are useful for debugging that issue -- your
> call on how useful they would be.

This actually is outside the scope of post_success (see indentation :)
This is the base mozharness run() method, so we're copying logs/* to build/upload/logs/* every run, even if it doesn't make sense (e.g. in cases where we're not doing anything with build/upload/, which is most jobs these days).

This doesn't actually turn it off, but gives us a way to turn it off, if we set copy_logs_post_run to False.

> ::: scripts/vcs-sync/vcs_sync.py
> @@ +781,5 @@
> >          self._write_repo_update_json(repo_map)
> >          if failure_msg:
> >              self.fatal("Unable to push these repos:\n%s" % failure_msg)
> >  
> > +    def preflight_upload(self):
> 
> This is a new method, but I don't see it being invoked anywhere in this
> changeset.
> 
> If this was a previously un-implemented method on the base class, shouldn't
> it include a super() call for safety? If not, where is it being invoked?

We by default run preflight_ACTION() if it exists, then ACTION() (with dashes converted to underscores), then postflight_ACTION() for all enabled actions, per http://hg.mozilla.org/build/mozharness/file/7d9425c91051/mozharness/base/script.py#l1051 .

Comment 85

[Aki Sasaki (not active)]

Comment on attachment 816264 [details] [diff] [review]
latest changes

http://hg.mozilla.org/build/mozharness/rev/be4c6ce194dc

Comment 86

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #81)
> Also, I think I'll close this bug and reopen new enhancement bugs after this
> lands.

Yay!
Also https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo