The default bug view has changed. See this FAQ.

automate beagle hg-git conversion in mozharness

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: aki, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozharness])

Attachments

(6 attachments)

(Assignee)

Description

4 years ago
Right now we have some scripts, a lot of config files, and a lot of docs+manual steps.

I want to get this config-driven and hands-off.
Also, I want the single git->git, single hg->hg, single hg->git, or single git->hg workflows to be a subset of this workflow.

This should handle both the gecko.git and beagle scenarios.

Let's see how realistic this is.
(Assignee)

Comment 1

4 years ago
Created attachment 721009 [details] [diff] [review]
check in gecko.git hgrc+config
Attachment #721009 - Flags: review?(hwine)
(Assignee)

Comment 2

4 years ago
Soft blocker: generic retries.
Soft blocker: GitVCS object (but my current approach will require it).
Depends on: 848227, 671345
(Assignee)

Comment 3

4 years ago
Progress here: https://github.com/escapewindow/mozharness/compare/vcs_conversion
(Assignee)

Updated

4 years ago
Depends on: 848908
Comment on attachment 721009 [details] [diff] [review]
check in gecko.git hgrc+config

Review of attachment 721009 [details] [diff] [review]:
-----------------------------------------------------------------

If these are the "as is", only comment is I'd prefer releases-gecko as the directory name to be consistent. I'd actually like to move the directory to that location in the current system to reduce the number of exceptions around it (I'd coordinate that).
Attachment #721009 - Flags: review?(hwine) → review+
(Assignee)

Comment 5

4 years ago
Comment on attachment 721009 [details] [diff] [review]
check in gecko.git hgrc+config

Pushed after an 'hg mv gecko releases-gecko' and commit.

http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/rev/adda5294a660
Attachment #721009 - Flags: checked-in+
(Assignee)

Updated

4 years ago
No longer depends on: 671345
(Assignee)

Updated

4 years ago
Depends on: 840305
(Assignee)

Comment 6

4 years ago
I've got wip in https://github.com/escapewindow/mozharness/tree/vcs_conversion that

* gets its source repo / target repo / branch list from a config file, rather than hardcodes or pre-populated .hg+.git directories
* avoids hg.m.o corruption by cloning/pulling into a clean source directory, then pulling from there into a working conversion directory.  If hg.m.o corrupts the source directory, we can blow that away and recreate without corrupting the working directory, which will save us conversion time.
* creates the work directory, populates, does the conversion

The script is currently converting m-c, which should take a few days.
I still need CVS prepending info from hwine before I'll have the right SHAs.
I'll need to add pushing logic and more error detection + verification, as well as notification.

In theory this script should be able to be fed different configs to create either beagle (gecko-dev.git) or the partner-oriented gecko.git.  I'll put that to the test later, after I get to CVS-based SHA parity.
(Assignee)

Comment 7

4 years ago
This is *still* converting ~10 days later, potentially because I pulled b2g18 in as well for testing purposes.  I either thought it would be an incremental increase or forgot about it (most likely both).

I'm going to look at gps' hg-git patches while this runs.
(Assignee)

Comment 8

4 years ago
* hwine's fork conversion is still running, but the shas appear to match so far
* latest upstream hg-git finished in less than a day, but the shas have diverged.

We're trying to figure out how to deal with this -- we may need to cherry pick gps' patches onto hal's fork, or something.
(Assignee)

Updated

4 years ago
Summary: automate all-in-one hg-git conversion in mozharness → automate beagle hg-git conversion in mozharness
(Assignee)

Updated

4 years ago
Duplicate of this bug: 730589
(Assignee)

Comment 10

4 years ago
(In reply to Aki Sasaki [:aki] from comment #8)
> * hwine's fork conversion is still running, but the shas appear to match so
> far
> * latest upstream hg-git finished in less than a day, but the shas have
> diverged.
> 
> We're trying to figure out how to deal with this -- we may need to cherry
> pick gps' patches onto hal's fork, or something.

Solved with https://github.com/escapewindow/hg-git , which is a new fork from latest upstream.
(Assignee)

Updated

4 years ago
Depends on: 869236
(Assignee)

Comment 11

4 years ago
I have cvs prepending basically working in automation, with the same shas as gecko.git.
(Assignee)

Comment 12

4 years ago
I now have the mapfile generated; it's fully a subset of gecko-mapfile.
(Assignee)

Comment 13

4 years ago
I noticed that I had made a *lot* of changes that are no longer needed (non-beagle changes, etc.) so I created a new branch and copied the needed 2 files over.

I had to remove the VCSConversionMixin import since that's no longer part of mozharness.base.vcs.vcsbase, but I wasn't using it in poc_beagle.py anyway.

https://github.com/escapewindow/mozharness/commits/beagle
(Assignee)

Comment 14

4 years ago
I was able to pull in mozilla-b2g18, and the mapfile is still fully a subset of gecko-mapfile.

Since I made a lot of changes since the first few actions were tested, and since we have a weekend, I made some backups and am now running a full conversion run from the beginning.
(Assignee)

Comment 15

4 years ago
The weekend run died at the end of prepend-cvs because it was looking for pre-cvs-mapfile at an old location (it worked on previous test runs because I hadn't clobbered the directory, so I had multiple copies of pre-cvs-mapfile lying around).

I fixed the location in the script, restarted the script from that point using the power of mozharness actions, and the script finished successfully.

The generated gecko-mapfile after pulling in m-c default and m-b2g18 default is fully a subset of git.m.o's gecko-mapfile.

Now I need to get push/upload/notify working, and the update loop error detection and recovery automated.  I'll need to be able to deal with the various not-so-edge cases.

I'm not sure how much we want to be able to bring up a new box from scratch via puppet; that's probably out of scope for this specific bug.
(Assignee)

Updated

4 years ago
Blocks: 799845
(Assignee)

Comment 16

4 years ago
The repo/branch update loop is now code complete, I think.

I now:

* update the virtualenv if new dependencies are added
* create a json file as describe in https://bugzilla.mozilla.org/show_bug.cgi?id=799845#c4
* upload to specified location(s)
* email specified people, every time or on failure only based on a flag

I need to deal with known error conditions, make this production-level bulletproof, test, and add the various other repos/branches.  I don't think I have tag support yet, and probably need to add that.
(Assignee)

Comment 17

4 years ago
(In reply to Aki Sasaki [:aki] from comment #16)
> I don't think
> I have tag support yet, and probably need to add that.

Added.
(Assignee)

Comment 18

4 years ago
(In reply to Aki Sasaki [:aki] from comment #17)
> (In reply to Aki Sasaki [:aki] from comment #16)
> > I don't think
> > I have tag support yet, and probably need to add that.
> 
> Added.

Sadly, while testing this, I noticed the <h<surkov email issue reared its ugly head again.  I'm going to have to investigate to see how widespread this issue is (my repo only, or also in Ehsan's github repo / Hal's conversion repo), why it's occurring, and how to either remove the issue or mitigate.
(Assignee)

Comment 19

4 years ago
(In reply to Aki Sasaki [:aki] from comment #18)
> Sadly, while testing this, I noticed the <h<surkov email issue reared its
> ugly head again.  I'm going to have to investigate to see how widespread
> this issue is (my repo only, or also in Ehsan's github repo / Hal's
> conversion repo), why it's occurring, and how to either remove the issue or
> mitigate.

I think this is fixed.  Running a new conversion to test.
(Assignee)

Comment 20

4 years ago
I don't think this is fixed, after the previous conversion.
Re-converting to right before the tag fixing, then *backing up*, and iterating on the steps after that point.
(Assignee)

Comment 21

4 years ago
I've been frustrated by my inability to get rid of the <h<surkov email commit.
I think the issue is I missed porting this line from the git-filter-branch docs:

for x in `git branch | grep -v ^*`; do git filter-branch -f -- "3229d5d8b7f8376cfb7936e7be810635a14a486b..$x"; if [ -d .git-rewrite/map ]; then cp -R .git-rewrite/map ../git-rewrite/map; fi; rm -rf .git-rewrite/; done

https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

Porting that, and retrying my prepend-cvs action once I nuke my conversion dir and restore from my pre-cvs backup.
(In reply to Aki Sasaki [:aki] from comment #21)
> I've been frustrated by my inability to get rid of the <h<surkov email
> commit.
> I think the issue is I missed porting this line from the git-filter-branch
> docs:
> 
> for x in `git branch | grep -v ^*`; do git filter-branch -f --
> "3229d5d8b7f8376cfb7936e7be810635a14a486b..$x"; if [ -d .git-rewrite/map ];
> then cp -R .git-rewrite/map ../git-rewrite/map; fi; rm -rf .git-rewrite/;
> done
> 
> https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

Yeah, you definitely do need that line.  Please let me know if there's something that I can do to help, most of this bug is sort of incomprehensible to my naked eyes.  :-)
(Assignee)

Comment 23

4 years ago
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #22)
> (In reply to Aki Sasaki [:aki] from comment #21)
> Yeah, you definitely do need that line.

Cool, good to know.  Hopefully after this finishes, I'll get rid of that <h<surkov once and for all.

> Please let me know if there's
> something that I can do to help, most of this bug is sort of
> incomprehensible to my naked eyes.  :-)

I CCed you mainly for informational purposes, since I *think* I un-stuck myself in comment 21.
(Assignee)

Comment 24

4 years ago
Latest error:

Halting on failure while running ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites', '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH']

I'm going to investigate later today.
(Assignee)

Comment 25

4 years ago
Hm.

https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion

"This repository has been disabled.

Access to this repository has been disabled by GitHub staff due to excessive use of resources. Contact support to restore access to this repository. Read here to learn more about decreasing the size of your repository. "
(In reply to comment #25)
> Hm.
> 
> https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion
> 
> "This repository has been disabled.
> 
> Access to this repository has been disabled by GitHub staff due to excessive
> use of resources. Contact support to restore access to this repository. Read
> here to learn more about decreasing the size of your repository. "

Hm, yeah I received an email about this on Saturday, and have asked the github support on what I can do but they haven't gotten back to me yet.  If you have suggestions on where else I can put it please let me know.  I have a slightly out of date bundle here <http://people.mozilla.org/~eakhgari/mozilla-history-tools.bundle> in case you just want to take a look at the instructions again.
(In reply to comment #26)
> (In reply to comment #25)
> > Hm.
> > 
> > https://github.com/ehsan/mozilla-history-tools/tree/master/initial_conversion
> > 
> > "This repository has been disabled.
> > 
> > Access to this repository has been disabled by GitHub staff due to excessive
> > use of resources. Contact support to restore access to this repository. Read
> > here to learn more about decreasing the size of your repository. "
> 
> Hm, yeah I received an email about this on Saturday, and have asked the github
> support on what I can do but they haven't gotten back to me yet.  If you have
> suggestions on where else I can put it please let me know.  I have a slightly
> out of date bundle here
> <http://people.mozilla.org/~eakhgari/mozilla-history-tools.bundle> in case you
> just want to take a look at the instructions again.

(and filed bug 878964 to see if I can host this on git.mozilla.org/)
(Assignee)

Comment 28

4 years ago
Aha,
20:25:22     INFO - Running command: ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites', '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH'] in /home/asasaki/moz2/build/conversion/beagle/.git
20:25:22     INFO - Copy/paste: /home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites -- 3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH
20:25:22     INFO - Using partial env: {'PATH': '%(PATH)s:/usr/libexec/git-core'}
20:25:22     INFO -  Cannot create a new backup.
20:25:22     INFO -  A previous backup already exists in refs/original/
20:25:22     INFO -  Force overwriting the backup with -f
20:25:22    ERROR - Return code: 1

I'm going to guess that relbranch was potentially already filter-branched via another branch or something.

I can either explicitly ignore this case or use -f.
Or blacklist this branch specifically from the git-filter-branch step.
(In reply to comment #28)
> Aha,
> 20:25:22     INFO - Running command:
> ['/home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites',
> '--', '3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH']
> in /home/asasaki/moz2/build/conversion/beagle/.git
> 20:25:22     INFO - Copy/paste:
> /home/asasaki/moz2/build/repo-sync-tools/git-filter-branch-keep-rewrites --
> 3ec464b55782fb94dbbb9b5784aac141f3e3ac01..COMM193a4_20100510_RELBRANCH
> 20:25:22     INFO - Using partial env: {'PATH':
> '%(PATH)s:/usr/libexec/git-core'}
> 20:25:22     INFO -  Cannot create a new backup.
> 20:25:22     INFO -  A previous backup already exists in refs/original/
> 20:25:22     INFO -  Force overwriting the backup with -f
> 20:25:22    ERROR - Return code: 1
> 
> I'm going to guess that relbranch was potentially already filter-branched via
> another branch or something.
> 
> I can either explicitly ignore this case or use -f.
> Or blacklist this branch specifically from the git-filter-branch step.

Hmm, I'm not sure if I remember hitting something similar to this when I was doing the conversion...
(Assignee)

Comment 30

4 years ago
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #29)
> (In reply to comment #28)
> > I'm going to guess that relbranch was potentially already filter-branched via
> > another branch or something.
> > 
> > I can either explicitly ignore this case or use -f.
> > Or blacklist this branch specifically from the git-filter-branch step.
> 
> Hmm, I'm not sure if I remember hitting something similar to this when I was
> doing the conversion...

I'm explicitly checking the status of each git-filter-branch-keep-rewrites call.
Your for loop in comment 21 could have hit this, and not errored out.  Unless you looked at the logs, you might have missed this.

It's also possible that I'm hitting something you hadn't, which I don't like the sound of.
(In reply to comment #30)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #29)
> > (In reply to comment #28)
> > > I'm going to guess that relbranch was potentially already filter-branched via
> > > another branch or something.
> > > 
> > > I can either explicitly ignore this case or use -f.
> > > Or blacklist this branch specifically from the git-filter-branch step.
> > 
> > Hmm, I'm not sure if I remember hitting something similar to this when I was
> > doing the conversion...
> 
> I'm explicitly checking the status of each git-filter-branch-keep-rewrites
> call.
> Your for loop in comment 21 could have hit this, and not errored out.  Unless
> you looked at the logs, you might have missed this.

Even though I don't have that explicit check, I'm fairly certain that I watched over the output logs pretty carefully.  But it's hard to say for sure given how long has passed since then...

> It's also possible that I'm hitting something you hadn't, which I don't like
> the sound of.

Are you running the initial conversion on the same mozilla-central changeset or a more recent one?
(Assignee)

Comment 32

4 years ago
I'm running it on tip.
I'd be surprised if COMM193a4_20100510_RELBRANCH was recent, though.
(Assignee)

Comment 33

4 years ago
COMM193a4_20100510_RELBRANCH isn't in https://github.com/mozilla/mozilla-central/branches , so something is different.
(In reply to comment #33)
> COMM193a4_20100510_RELBRANCH isn't in
> https://github.com/mozilla/mozilla-central/branches , so something is
> different.

Are you sure?

$ git branch -a | grep COMM193a4
  remotes/mc/COMM193a4_20100510_RELBRANCH
  remotes/origin/COMM193a4_20100510_RELBRANCH
(Assignee)

Comment 35

4 years ago
Ok. I still don't see it in the html page though.
(Assignee)

Comment 36

4 years ago
Sorry, buried in the GECKO branches. N/m me.
(In reply to comment #36)
> Sorry, buried in the GECKO branches. N/m me.

Yeah... I don't find github's UI particularly useful for navigating branches...  (and tags for that matter!)  You're definitely not the first person confused by it.  :-)
(Assignee)

Comment 38

4 years ago
Tl;dr status update: I'm still working on this.


I'm still trying to verify that git-filter-branch-keep-rewrites will work with -f.

I've hit the following:
* prepended cvs to a restored backup of the conversion dir that only had the master branch
* blew that away, started from scratch, backed up the conversion dir before prepending cvs
* hit the above -f issue
* restored the conversion dir, restarted cvs prepend, forgot to git pull my latest changes (-f), so hit the above -f issue again
* restored the conversion dir, restarted cvs prepend, died on the mc-git-rewrite dir existing when trying to move the .git-rewrite dir out.
** added an rmtree, and fixed the move() method
* tried running the git-filter-branch-keep-rewrites -f manually, but this was taking a while
** tried running the automated cvs prepend simultaneously, ran dangerously close to 100% inode usage
** killed manual git-filter-branch-keep-rewrites, let automated cvs prepend run

I'm currently here.  I can investigate other things in parallel once bug 877922 is fixed; until then I'm stuck with single-threaded testing.

Hoping this works, so I can then a) backup the conversion dir and mc-git-rewrite dir again, b) fix tags, and c) verify that <h<surkov is gone.
(Assignee)

Comment 39

4 years ago
The git-filter-branch bit finished with no errors! -f worked.
I need to fix tags, then verify <h<surkov is gone.  I've already backed up the dirs so I can get back to this point without re-converting, assuming I haven't done anything wrong.

Will get back to this around buildduty emergencies.
(Assignee)

Comment 40

4 years ago
Good news: I was able to push to all my test repos without issue, which tells me <h<surkov may not be an issue anymore!

Bad news: my tip revision is unknown in gecko.git, meaning the shas have diverged somewhere :( :(

Tracking this down.  I currently suspect the git-filter-branch-keep-rewrites -f of an old m-c branch changed a parent revision of master somewhere.
Does gecko.git have the CVS history?
(Assignee)

Comment 42

4 years ago
Yes, it does.  My conversion was fully a subset of gecko.git's shas until I started adding the other mozilla-central branches.
(In reply to comment #42)
> Yes, it does.  My conversion was fully a subset of gecko.git's shas until I
> started adding the other mozilla-central branches.

If you take one of those branches and do a git merge-base master <branch-name>, what do you get?  That's the best way to find out where histories deviate.
(Assignee)

Comment 44

4 years ago
Hm,
git merge-base master COMM193a4_20100510_RELBRANCH
2249d7ec3a12c82bf01e787fffb8e7b12f9cc9a6

I'm currently suspecting that the -f wasn't the right approach, and I should be ignoring errors.  I'm going to try that approach while also trying to debug the sha divergence in a backup directory.
(Assignee)

Comment 45

4 years ago
I re-ran conversion without the -f.

1. Two things:

a) It's a lot faster, but the <h<surkov email is back if I run git-filter-branch-keep-rewrites without -f.
b) The shas still diverge, which means the -f wasn't the [sole] culprit in the shas diverging.


2. Potential reasons the shas might have diverged:

a) When I blew away and recreated everything, it's possible one of the python packages changed versions.  I should investigate and lock down these versions.

b) It's possible that since I'm converting a larger number of commits than I was previously, a newer commit reacted differently to an initial conversion step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it actively munges emails, etc.)

c) It's possible that the act of initially converting other m-c branches other than default by itself results in different shas than we have in gecko.git, which only has m-c default + various b2g18 branches.  If this is true we are probably in a bad spot.

d) Something else I haven't thought of.


3. Things I can try:

a) I forgot to compare the pre-cvs shas against Hal's pre-cvs releases-mozilla-central git repo during the past two conversions.  If these differ, then we've isolated the sha divergence to pre-git-filter-branch; if not, we've ruled that out.  I'm untarring my backup and doing that.

b) Hal walked me through finding sha divergence earlier (after comment 8) but I've forgotten how.  I think I may need help here again.


Needinfo?ing for 3b.
Flags: needinfo?(hwine)
Flags: needinfo?(ehsan)
(Assignee)

Comment 46

4 years ago
(In reply to Aki Sasaki [:aki] from comment #45) 
> 3. Things I can try:
> 
> a) I forgot to compare the pre-cvs shas against Hal's pre-cvs
> releases-mozilla-central git repo during the past two conversions.  If these
> differ, then we've isolated the sha divergence to pre-git-filter-branch; if
> not, we've ruled that out.  I'm untarring my backup and doing that.

I just did a |git log| inside my restored pre-cvs backup and verified that the newest sha existed inside Hal's pre-cvs releases-mozilla-central git repo.

The restored pre-cvs backup also has an expected sha of 2514a423aca5d1273a842918589e44038d046a51 for the "Free the (distributed) Lizard!" commit.

The sha divergence comes during or after the cvs-prepend step.
(In reply to comment #45)
> a) It's a lot faster, but the <h<surkov email is back if I run
> git-filter-branch-keep-rewrites without -f.
> b) The shas still diverge, which means the -f wasn't the [sole] culprit in the
> shas diverging.

:(

> 2. Potential reasons the shas might have diverged:
> 
> a) When I blew away and recreated everything, it's possible one of the python
> packages changed versions.  I should investigate and lock down these versions.

Which python packages?

> b) It's possible that since I'm converting a larger number of commits than I
> was previously, a newer commit reacted differently to an initial conversion
> step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> actively munges emails, etc.)

The only way that would be possible is if hg-git was modified after the previous initial conversion, which it was.  The only thing that can affect the sha1 of a commit based on the parent(s) is the sha1 for that parent commit.

> c) It's possible that the act of initially converting other m-c branches other
> than default by itself results in different shas than we have in gecko.git,
> which only has m-c default + various b2g18 branches.  If this is true we are
> probably in a bad spot.

I don't see how this could possibly be the case.

> d) Something else I haven't thought of.
> 
> 3. Things I can try:
> 
> a) I forgot to compare the pre-cvs shas against Hal's pre-cvs
> releases-mozilla-central git repo during the past two conversions.  If these
> differ, then we've isolated the sha divergence to pre-git-filter-branch; if
> not, we've ruled that out.  I'm untarring my backup and doing that.
> 
> b) Hal walked me through finding sha divergence earlier (after comment 8) but
> I've forgotten how.  I think I may need help here again.

git merge-base?  Or are you asking how to use merge-base?
(Assignee)

Comment 48

4 years ago
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #47)
> (In reply to comment #45)
> > 2. Potential reasons the shas might have diverged:
> > 
> > a) When I blew away and recreated everything, it's possible one of the python
> > packages changed versions.  I should investigate and lock down these versions.
> 
> Which python packages?

01:00:38     INFO - Current package versions:
01:00:38     INFO -   mapper == 0.1
01:00:38     INFO -   ordereddict == 1.1
01:00:38     INFO -   dulwich == 0.9.0
01:00:38     INFO -   bottle == 0.11.6
01:00:38     INFO -   distribute == 0.6.28
01:00:38     INFO -   hg-git == 0.4.0-moz2
01:00:38     INFO -   mercurial == 2.2.1

> > b) It's possible that since I'm converting a larger number of commits than I
> > was previously, a newer commit reacted differently to an initial conversion
> > step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> > actively munges emails, etc.)
> 
> The only way that would be possible is if hg-git was modified after the
> previous initial conversion, which it was.  The only thing that can affect
> the sha1 of a commit based on the parent(s) is the sha1 for that parent
> commit.

I think the sha differs if you change <h<surkov to <h?surkov or <hsurkov , no?

Basically, git-filter-branch changes things in the commits, like timestamps and email addresses.  You ran git-filter-branch against N number of commits, Hal ran it later against N + N1 number of commits, I ran it later against N + N1 + N2 number of commits.  This time I ran it against N + N1 + N2 + N3 number of commits, and I'm hypothesizing that git-filter-branch munged something in that last N3 batch of commits to make my shas no longer match.

> > c) It's possible that the act of initially converting other m-c branches other
> > than default by itself results in different shas than we have in gecko.git,
> > which only has m-c default + various b2g18 branches.  If this is true we are
> > probably in a bad spot.
> 
> I don't see how this could possibly be the case.

Ok.

> > b) Hal walked me through finding sha divergence earlier (after comment 8) but
> > I've forgotten how.  I think I may need help here again.
> 
> git merge-base?  Or are you asking how to use merge-base?

We're hypothesizing you mean a) pull gecko.git into my working conversion dir and b) try to see the common ancestor between the two master revisions?

The method Hal showed me before is

git --git-dir build/conversion/beagle/.git log --format="%H %P %ai %s" --reverse --topo-order > shas

And do the same thing with gecko.git, then diff the sha text files.  The first line that diffs should be the first divergent sha.

I'll try this method first, since it's less destructive to my working dir, and I'm already short on disk+inodes.
Flags: needinfo?(hwine)
(In reply to comment #48)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #47)
> > (In reply to comment #45)
> > > 2. Potential reasons the shas might have diverged:
> > > 
> > > a) When I blew away and recreated everything, it's possible one of the python
> > > packages changed versions.  I should investigate and lock down these versions.
> > 
> > Which python packages?
> 
> 01:00:38     INFO - Current package versions:
> 01:00:38     INFO -   mapper == 0.1
> 01:00:38     INFO -   ordereddict == 1.1
> 01:00:38     INFO -   dulwich == 0.9.0
> 01:00:38     INFO -   bottle == 0.11.6
> 01:00:38     INFO -   distribute == 0.6.28
> 01:00:38     INFO -   hg-git == 0.4.0-moz2
> 01:00:38     INFO -   mercurial == 2.2.1

I think only hg-git matters in this list.

> > > b) It's possible that since I'm converting a larger number of commits than I
> > > was previously, a newer commit reacted differently to an initial conversion
> > > step than expected.  (I would suspect git-filter-branch-keep-rewrites, as it
> > > actively munges emails, etc.)
> > 
> > The only way that would be possible is if hg-git was modified after the
> > previous initial conversion, which it was.  The only thing that can affect
> > the sha1 of a commit based on the parent(s) is the sha1 for that parent
> > commit.
> 
> I think the sha differs if you change <h<surkov to <h?surkov or <hsurkov , no?

Yes.  git commits are basically plaintext files of the following format:

tree f7722149a10a1347075520160b3cbd1ef4ef6a40
parent 062746d6f8015f68f72a593bdee720955f71ffcf
author Ehsan Akhgari <ehsan@mozilla.com> 1371518325 -0400
committer Ehsan Akhgari <ehsan@mozilla.com> 1371681249 -0400

Commit message

The commit sha1 is the sha1 of this text file.  What I meant to say is that the only thing from the parent commit that affects the sha1 of this commit is the sha1 of the parent commit.  So if uyyou do anything to change the sha1 of 1 commit, everything will diverge from that point on.

> Basically, git-filter-branch changes things in the commits, like timestamps and
> email addresses.  You ran git-filter-branch against N number of commits, Hal
> ran it later against N + N1 number of commits, I ran it later against N + N1 +
> N2 number of commits.  This time I ran it against N + N1 + N2 + N3 number of
> commits, and I'm hypothesizing that git-filter-branch munged something in that
> last N3 batch of commits to make my shas no longer match.

If the merge base is within N3, then you're right.

> > > b) Hal walked me through finding sha divergence earlier (after comment 8) but
> > > I've forgotten how.  I think I may need help here again.
> > 
> > git merge-base?  Or are you asking how to use merge-base?
> 
> We're hypothesizing you mean a) pull gecko.git into my working conversion dir
> and b) try to see the common ancestor between the two master revisions?

Yes.

> The method Hal showed me before is
> 
> git --git-dir build/conversion/beagle/.git log --format="%H %P %ai %s"
> --reverse --topo-order > shas
> 
> And do the same thing with gecko.git, then diff the sha text files.  The first
> line that diffs should be the first divergent sha.

I believe this should result in the same thing, it will only be slower...  But I haven't tested this.  git-merge-base is how git finds the merge base for any two commits, and is going to be way less error prone.
(Assignee)

Comment 50

4 years ago
May 19.

< 8962237ccb5c517d8a63ac681c89e758afda3bdb d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test
> 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test

From gecko.git:

commit 8962237ccb5c517d8a63ac681c89e758afda3bdb
Author: Carlos G. <charlie.brown.uy@gmail.com>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test


From my conversion dir:

commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646
Author: Carlos G <charlie.brown.uy@gmail.com>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test



Comment 15, where my conversion dir was fully a subset of gecko.git, was May 13.
My theory is that git-filter-branch-keep-rewrites is stripping the trailing '.' from "Carlos G."

I may have to limit the initial conversion to stop before this revision, git-filter-branch-keep-rewrites with -f, and then convert the rest with just hg-git.

Not entirely sure how to find the parent revision's hg revision, but I'll figure it out.
(Assignee)

Comment 51

4 years ago
Hm, I should verify the new version of hg-git (0.4.0-moz2) isn't munging this before I go through the whole |git-filter-branch-keep-rewrites -f| multi-day conversion.
(In reply to Aki Sasaki [:aki] from comment #50)
> May 19.
> 
> < 8962237ccb5c517d8a63ac681c89e758afda3bdb
> d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug
> 463696 - crash test
> > 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 d5e239d8e2a8218db85c1c62da78150157dbf834 2013-05-19 18:20:20 +0000 Bug 463696 - crash test
> 
> From gecko.git:
> 
> commit 8962237ccb5c517d8a63ac681c89e758afda3bdb
> Author: Carlos G. <charlie.brown.uy@gmail.com>
> Date:   Sun May 19 18:20:20 2013 +0000
> 
>     Bug 463696 - crash test
> 
> 
> From my conversion dir:
> 
> commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646
> Author: Carlos G <charlie.brown.uy@gmail.com>
> Date:   Sun May 19 18:20:20 2013 +0000
> 
>     Bug 463696 - crash test
> 
> 
> 
> Comment 15, where my conversion dir was fully a subset of gecko.git, was May
> 13.
> My theory is that git-filter-branch-keep-rewrites is stripping the trailing
> '.' from "Carlos G."

That seems like a bug. :(

> I may have to limit the initial conversion to stop before this revision,
> git-filter-branch-keep-rewrites with -f, and then convert the rest with just
> hg-git.
> 
> Not entirely sure how to find the parent revision's hg revision, but I'll
> figure it out.

Just look in the git-mapfile.
Flags: needinfo?(ehsan)
(Assignee)

Comment 53

4 years ago
(In reply to Aki Sasaki [:aki] from comment #51)
> Hm, I should verify the new version of hg-git (0.4.0-moz2) isn't munging
> this before I go through the whole |git-filter-branch-keep-rewrites -f|
> multi-day conversion.

In my pre-cvs-prepend backup, the '.' is kept:

commit 7bcdb26c4e194e80744f4beb76f2df58b4bb475e
Author: Carlos G. <charlie.brown.uy@gmail.com>
Date:   Sun May 19 18:20:20 2013 +0000

    Bug 463696 - crash test

That points the finger directly at the cvs-prepend steps, primarily git-filter-branch-keep-rewrites.
(Assignee)

Comment 54

4 years ago
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #52)
> (In reply to Aki Sasaki [:aki] from comment #50)
> > I may have to limit the initial conversion to stop before this revision,
> > git-filter-branch-keep-rewrites with -f, and then convert the rest with just
> > hg-git.
> > 
> > Not entirely sure how to find the parent revision's hg revision, but I'll
> > figure it out.
> 
> Just look in the git-mapfile.

This is the problem commit:
http://hg.mozilla.org/mozilla-central/rev/82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2

The theory is, if I limit the clone for the initial conversion to stop at its parent ( http://hg.mozilla.org/mozilla-central/rev/2701855384d6 ) then git-filter-branch won't munge this commit, and subsequent hg-git passes will do the right thing, and shas won't diverge.
(Assignee)

Comment 55

4 years ago
(In reply to Aki Sasaki [:aki] from comment #54)
> This is the problem commit:
> http://hg.mozilla.org/mozilla-central/rev/
> 82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2
> 
> The theory is, if I limit the clone for the initial conversion to stop at
> its parent ( http://hg.mozilla.org/mozilla-central/rev/2701855384d6 ) then
> git-filter-branch won't munge this commit, and subsequent hg-git passes will
> do the right thing, and shas won't diverge.

I think I can clone, then use hg strip:
http://mercurial.selenic.com/wiki/PruningDeadBranches#Using_strip

hg --config extensions.mq= strip -n 82e4f1b7bbb6e30a635b49bf2107b41a8c26e3d2
(Assignee)

Comment 56

4 years ago
Verified that the same commit 66eddbf4c3bf738bc63ba7f4fcc9dba99ed51646 exists in the |git-filter-branch-keep-rewrites -f| backup, and that the final tip-of-master sha exists in my current working directory.

I think if I do the above |hg strip|, and run git-filter-branch-keep-rewrites with -f, I will fix both the sha divergence and the <h<surkov email problem.

I'm going to start that today; judging by the last attempt, this may take til mid-next week.
(Assignee)

Comment 57

4 years ago
Converted, prepended cvs, fixed tags.
As of right now, the tip of master's sha exists in gecko.git (non-divergent shas for master branch).

I'm going to back up the working dir and then do a conversion loop get the rest of may 16 -> today converted + pushed.  (The push is also where we detect the absence or existence of the <h<surkov email.)
(Assignee)

Comment 58

4 years ago
...aaaaaaand we have both divergent shas and <h<surkov.

My current hypothesis is the hg strip left behind some unstripped revisions that weren't "tip of master", but children of tip-of-master.  So they weren't git-filter-branched and screwed things up.  If this is the case, nuking them from orbit before the push loop (or maybe before the tag fixing, or maybe before the cvs prepending, depending) may fix things.

I will debug in a bit.
(Assignee)

Comment 59

4 years ago
<h<surkov:

* The offending revision is b294eb35d142755f7724ed5bf96f11d2d4054a1b
* |git branch --contains b294eb35d142755f7724ed5bf96f11d2d4054a1b| only gives "master"
** I had already started to wonder if my |hg strip| was faulty, since "default" had a number of children.  I'm guessing I need to strip further up so tip-of-default has zero children before I start the initial conversion.

Diverging shas:
* I have two separate "Free the (distributed) Lizard!" commits in my conversion dir.
** One is 05e5d33a570d48aed58b2d38f5dfc0a7870ff8d3 , which matches gecko.git
*** |git branch --contains 05e5d33a570d48aed58b2d38f5dfc0a7870ff8d3| gives me a lot of (all?) branches, including a "* master".
*** This revision shows in my post-cvs-mapfile and gecko-mapfile, but not the pre-cvs-mapfile
** The other is 2514a423aca5d1273a842918589e44038d046a51 , which doesn't exist in gecko.git
*** |git branch --contains 2514a423aca5d1273a842918589e44038d046a51| gives "master"
*** This revision shows in my pre-cvs-mapfile, but not the post-cvs-mapfile or gecko-mapfile

I traced http://hg.mozilla.org/mozilla-central/rev/82e4f1b7bbb6 's children until the merge -- I think the issue is the graph doesn't allow for cleanly stripping that single revision (maybe?): http://hg.mozilla.org/mozilla-central/graph/132363

I think keeping the same approach, but stripping all revisions that came after http://hg.mozilla.org/mozilla-central/rev/317fe0f314ab might be one approach.  Verifying this via re-converting will be time consuming, so I want to do some more digging first, but I currently don't see anything glaring to prove this line of thinking wrong.
(Assignee)

Comment 60

4 years ago
After I hg strip these revisions: ("26cb30a532a1", "aad29aa89237", "eb1d7f3cd1d7", "9f2fa4839e98"), revision 317fe0f314ab shows up as 'tip', and it appears to have no child revisions:

deathduck:/src/vcs_conversion/hg-mc [15:52:44] (default)
534$ hg log -r 132332
changeset:   132332:317fe0f314ab
tag:         tip
parent:      132331:1196b497640b
parent:      132290:8ca260fe91e3
user:        Phil Ringnalda <philringnalda@gmail.com>
date:        Sat May 18 18:07:46 2013 -0700
summary:     Merge the last PGO-green cset from m-i to m-c

deathduck:/src/vcs_conversion/hg-mc [15:52:48] (default)
535$ hg log -r 132333
abort: unknown revision '132333'!

An |hg glog| appears to confirm.  I'm going to pull backups of this conversion run to gd3 for later investigation, if desired, then I'm going to start another conversion run with the new hg strip logic.
(Assignee)

Comment 61

4 years ago
Status:

I started off a conversion run after comment 60.  The initial conversion went smoothly; I backed up the conversion directory afterwards, and verified that the tip-of-master sha existed in the releases-mozilla-central github repo (this repo doesn't have CVS history, so the shas should match).  Good.

Afterwards, I kicked off the cvs prepend step, and crossed my fingers, since this would [dis?]prove my hg strip theory.  And I let it run, as the variopus git-filter-branch-keep-rewrites steps for all the m-c branches combine for around a week of compute time.

On Wednesday, the machine rebooted to resolve bug 894225.  I lost nothing that I couldn't reproduce, but I did lose 4 days of prepend-cvs compute time.  I didn't have any way of restarting the prepend-cvs step midway, and didn't trust that either.  I restored my initial conversion backup, wrote a patch to run incremental backups of the conversion dir after each branch, then restarted the prepend-cvs step yesterday.

With any luck, this will finish late next week and we'll be able to verify whether the hg strip fixes result in a good conversion.
(Assignee)

Comment 62

4 years ago
...... I think we're good!!

This finished, and pushed to github without issue (this is how I caught the <h<surkov issue before).

The latest changeset on master, 9a1a3a0658790de7f95e0ab4a18ba78f71266b96, exists on gecko.git:
http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=9a1a3a0658790de7f95e0ab4a18ba78f71266b96

I only pushed the master/b2g18{,_v_1_0_0,_v_1_0_1} branches to https://github.com/escapewindow/test-beagle , but it's there for people to look at.

I still need to:

* add the rest of the branches that we want to be a part of this repo
* relatively minor code cleanup
* split the patch up into ~5 patches for easier review

for the script portion.  Automating running this script should be relatively easy, as well as questions like bug 847640.
(Assignee)

Comment 63

4 years ago
While populating some of the rest of the branches, I noticed that a number of the FIREFOX_/FENNEC_ etc. tags aren't getting populated.
https://github.com/escapewindow/test-beagle/branches

If I were only supporting beagle, I could potentially pull in all branches and tags, and allow for moving tags (releng release automation regularly moves {FIREFOX,FENNEC}_*_RELEASE tags; git doesn't like tag moves).  However, I've been keeping an eye out for supporting the partner-oriented gecko.git as well, in which case moving tags are verboten.

I probably need to change my repo/branch/tag definitions:
https://github.com/escapewindow/mozharness/blob/beagle/configs/users/aki/beagle1.py#L50

* branches and tags need:
** incoming config (superset of all branches+tags needed), with whitelist, blacklist?, regexes, and potential renames.
** outgoing config, which can be global or per-target, with whitelist, blacklist?, regexes, and potential renames.
* ideally some sort of config verification, so we don't make a config change to the production process that pushes bad changes to public repos
* possibly make it easier to deal with than creating one massive dict

The feedback loop in testing this should be massively faster than the cvs prepend feedback loop, though, so this should be a relatively fast change to make.


Also, I'm going to write a blog post about my work here in the next couple days.
(Assignee)

Updated

4 years ago
Depends on: 892563
No longer depends on: 840305
(Assignee)

Updated

4 years ago
Depends on: 898711
(Assignee)

Updated

4 years ago
Depends on: 898714
(Assignee)

Updated

4 years ago
Depends on: 898715
(Assignee)

Updated

4 years ago
Depends on: 898716
(Assignee)

Comment 64

4 years ago
http://escapewindow.dreamwidth.org/238476.html
Product: mozilla.org → Release Engineering
(Assignee)

Updated

4 years ago
Depends on: 908369
(Assignee)

Comment 65

4 years ago
Merged my beagle_branches branch into the beagle branch, then got this:



02:48:48    ERROR -  abort: Parent SHA-1 not present in Gitrepo: 774108fb3240461699cd5390a08bfe8c75f8768d
02:48:48    ERROR -  Automation Error: hg not responding
02:48:49    ERROR - Return code: 65280
02:49:04    ERROR -  error: src refspec refs/heads/GECKO140_2012061213_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE80_2011102618_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE70_2011083009_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE150_2012071710_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO90_2011110909_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO150_2012082116_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO150_2012082118_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO100_2012011108_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/MOBILE90_2011121217_RELBRANCH does not match any.
02:49:04    ERROR -  error: src refspec refs/heads/GECKO190_2013010816_RELBRANCH does not match any.
02:49:04    ERROR - Return code: 256
02:49:04    ERROR - mozilla-beta: Can't push /home/asasaki/beagle1/build/conversion/beagle to /home/asasaki/beagle1/build/target/beagle/.git!
<snip>

Looks like I'll need a real fix for bug 908369, not just my overwrite workaround.
(Assignee)

Comment 66

4 years ago
Found it, I think.

lrwxrwxrwx 1 asasaki asasaki       107 Jul 18 01:44 pack-0f19f543bfed791cc99f3168547e28cd86598c3e.idx -> /home/asasaki/moz2/build/mozilla-cvs-history/objects/pack/pack-0f19f543bfed791cc99f3168547e28cd86598c3e.idx
lrwxrwxrwx 1 asasaki asasaki       108 Jul 18 01:44 pack-0f19f543bfed791cc99f3168547e28cd86598c3e.pack -> /home/asasaki/moz2/build/mozilla-cvs-history/objects/pack/pack-0f19f543bfed791cc99f3168547e28cd86598c3e.pack

On gd2, I was in /home/asasaki/moz2/.  On gd3, I'm in /home/asasaki/beagle1/.
When I rsynced over the conversion dir, these absolute path softlinks were pointing to nothing.

When I softlinked beagle1 to moz2, this started working again.  Verifying after this next conversion pass.
(Assignee)

Comment 67

4 years ago
Created attachment 796368 [details] [diff] [review]
beagle.diff

56kb, and months of changes.
Let me know if you have any questions; thanks in advance :)
Attachment #796368 - Flags: review?(rail)
Comment on attachment 796368 [details] [diff] [review]
beagle.diff

Review of attachment 796368 [details] [diff] [review]:
-----------------------------------------------------------------

I didn't test it (!), but in overall it looks good.

See some minor suggestions below.

::: configs/vcs_sync/beagle.py
@@ +13,5 @@
> +config = {
> +    "log_name": "beagle",
> +    "log_max_rotate": 99,
> +    "repos": [{
> +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",

Should we move these repos to /build?

@@ +43,5 @@
> +                "default": "master",
> +            },
> +        },
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,

the final production version should use something different

@@ +46,5 @@
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> +    "cvs_manifest": CVS_MANIFEST,
> +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",

The same here. Maybe something that could be publicly accessible?

::: scripts/hg_git.py
@@ +938,5 @@
> +        for notify_config in c.get('notify_config', []):
> +            if not fatal and notify_config.get('failure_only'):
> +                continue
> +            fromaddr = notify_config.get('from', c['default_notify_from'])
> +            message = string.join((

string is deprecated, use "\r\n".join(<...>) instead?
Attachment #796368 - Flags: review?(rail) → review+
(Assignee)

Comment 69

4 years ago
Comment on attachment 796368 [details] [diff] [review]
beagle.diff

https://hg.mozilla.org/build/mozharness/rev/de02a1b7786d

Woot!

I did fix the string -> '\r\n'.join().  The user-specific stuff is in there for test-beagle; we should fix when we go to real beagle.
Attachment #796368 - Flags: checked-in+
(Assignee)

Updated

4 years ago
Depends on: 857696
(Assignee)

Updated

4 years ago
Depends on: 922317
(Assignee)

Updated

4 years ago
Depends on: 924716
(Assignee)

Updated

4 years ago
Depends on: 924719
(Assignee)

Comment 70

4 years ago
Created attachment 815544 [details] [diff] [review]
base.diff

part 1 of ?

This patch:

* gets rid of some print's in mozharness.base.config
* updates GitErrorList
* fixes a virtualenv log line
* removes a duplicate copy/paste log line from get_output_from_command()
* allows add_failure() to not change the exit status (return_code)
* some whitespace tweaks
Attachment #815544 - Flags: review?(rail)
Attachment #815544 - Flags: review?(rail) → review+
(Assignee)

Comment 71

4 years ago
Created attachment 815555 [details] [diff] [review]
script_cleanup

Not a required patch, just cleanup I did while I was in here.

* chmod 755 scripts/*.py
* removes obsolete device_talosrunner.py (obsoleted by android_panda_talos.py) and unfinished/old mozmill_updates.py
Attachment #815555 - Flags: review?(rail)
(Assignee)

Comment 72

4 years ago
Created attachment 815574 [details] [diff] [review]
vcs_sync

Sorry about this.

Rail for code review, hwine for feedback about configs, if they're understandable without grokking the code.

This patch is currently staging-oriented, but the production changes should be minimal.
Let me know if there's anything I can do to make this review easier.

* renames the github ssh key since it's currently an additional throwaway key that I added to my github account.
* points at the new local pypi servers
* adds gecko-git, l10n, and project-branches configs.  The l10n config is skippable for this commit if there are concerns.
* creates a shared VCSSyncScript object that currently only deals with notification.  I wanted to avoid touching the initial_beagle.py script too much, since testing changes requires a week of compute time.
* splits hg_git.py into vcs-sync/initial_beagle.py and vcs-sync/vcs_sync.py so changes to the evolving vcs_sync.py don't put initial_beagle.py at too much risk.
* initial_beagle
** fixes a stupid bug that made us skip any branches left over (if the branches lenght wasn't divisible by 10)
** gets rid of the update-/push/upload steps and helper methods from initial_beagle, since we don't need them.
** hg strip's another errant head that caused <h<surkov to appear again; adds a check to fatal() if 317fe0f314ab is not the only m-c head so we don't waste more time in the future
* vcs_sync.py
** gets rid of the pull/create-/initial-conversion/prepend-cvs/fix-tags steps, since those aren't needed in the update loop
** adds support for multiple conversion dirs (beagle+gecko.git all happen in one; build/ and l10n repos want their own conversion dirs)
** adds support for defining locales and project branches by name, and generates the conversion repo dicts for you (to make it easier for us)
** makes the loop noop for a repo if there are no incoming changes.  This cuts our noop time from ~550 seconds to ~20 seconds (dependent on hg.m.o load)
** adds mapfile combining support, but doesn't use it yet
** adds first-time support to update_work_mirror() since the create_work_mirror() method was removed
** adds more HgErrorList checking
** runs gexport per repo, rather than one time for the entire conversion dir (needed for multi-conversion-dirs)
** allows for mapfile renaming per repo
Attachment #815574 - Flags: review?(rail)
Attachment #815574 - Flags: feedback?(hwine)
(Assignee)

Comment 73

4 years ago
Comment on attachment 815544 [details] [diff] [review]
base.diff

http://hg.mozilla.org/build/mozharness/rev/67d0c83ee405
Attachment #815544 - Flags: checked-in+
Comment on attachment 815574 [details] [diff] [review]
vcs_sync

Review of attachment 815574 [details] [diff] [review]:
-----------------------------------------------------------------

gecko-git.py matches what the legacy config shows

Everything that I can easily check looks okay.

I'd love to see some additional documentation on the configs -- whether that's like a sample config with comments (ala Apache) or lighter weight annotations in each config file.

Also, for the future, it'd be great to have a config sanity checker -- it sounds like there are some "shouldn't go together", and shadowed configs that could lead to confusion years from now.

::: configs/vcs_sync/gecko-git.py
@@ +1,4 @@
> +import os
> +import socket
> +hostname = socket.gethostname()
> +

I don't see any mapfile_name specified here, but we need a mapfile. Where is that handled?

@@ +13,5 @@
> +config = {
> +    "log_name": "gecko-git",
> +    "log_max_rotate": 99,
> +    "repos": [{
> +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",

Curious as to what is being used from this repo. It may make sense to get it somewhere more permanent if things are going to live on.

@@ +41,5 @@
> +    },
> +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> +    "cvs_manifest": CVS_MANIFEST,
> +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",

Shouldn't the initial artifacts be in tooltool?

@@ +96,5 @@
> +            "branches": {
> +                "default": "gecko-18",
> +            },
> +        },
> +        "tag_config": {

This tag_config is redundant, as both targets already have one specified.

Not a functional issue, but it can leave humans confused about which wins :)

::: configs/vcs_sync/l10n.py
@@ +1,5 @@
> +from copy import deepcopy
> +import os
> +import socket
> +hostname = socket.gethostname()
> +

I don't see any mapfile specifications here, but we do need one integrated mapfile for all l10n.

How is that handled?

::: configs/vcs_sync/project-branches.py
@@ +7,5 @@
> +PROJECT_BRANCHES = [
> +    # twig projects
> +    "ash",
> +    "alder",
> +    "ash",

duplicate twig "ash"

::: mozharness/base/vcs/vcssync.py
@@ +11,5 @@
> +import smtplib
> +import sys
> +import time
> +
> +sys.path.insert(1, os.path.dirname(os.path.dirname(os.path.dirname(sys.path[0]))))

Wow! that's cryptic. Is this a mozharnessism?

@@ +66,5 @@
> +                "To: %s" % notify_config['to'],
> +                "CC: %s" % ','.join(notify_config.get('cc', [])),
> +                "Subject: %s" % subject,
> +                "",
> +                text

SMTP lawyer here -- the body of the message is also supposed to use network format \r\n, but 'text' is built using only '\n'

In this day and age, it'll probably work, but :)
Attachment #815574 - Flags: feedback?(hwine) → feedback+
(Assignee)

Comment 75

4 years ago
(In reply to Hal Wine [:hwine] (use needinfo) from comment #74)
> gecko-git.py matches what the legacy config shows
>
> Everything that I can easily check looks okay.

Cool, thanks!

> I'd love to see some additional documentation on the configs -- whether
> that's like a sample config with comments (ala Apache) or lighter weight
> annotations in each config file.

Sure.

> Also, for the future, it'd be great to have a config sanity checker -- it
> sounds like there are some "shouldn't go together", and shadowed configs
> that could lead to confusion years from now.

Sounds like bug 699343, which would be awesome to fix, but I don't know where to start.
And/or completely pulling the conversion_repos portion out of the config file into a separate file that the config file references.  But the current format is my best effort at this point in time for the format.  Maybe when the back end is explained more it'll make more sense.

> ::: configs/vcs_sync/gecko-git.py
> @@ +1,4 @@
> > +import os
> > +import socket
> > +hostname = socket.gethostname()
> > +
>
> I don't see any mapfile_name specified here, but we need a mapfile. Where is
> that handled?

It defaults to gecko-mapfile in the code; it's only overridden in the configs.
If we had apache-style comments we might have gecko-mapfile in the config, but commented out.

> @@ +13,5 @@
> > +config = {
> > +    "log_name": "gecko-git",
> > +    "log_max_rotate": 99,
> > +    "repos": [{
> > +        "repo": "https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools",
>
> Curious as to what is being used from this repo. It may make sense to get it
> somewhere more permanent if things are going to live on.

As mentioned in IRC, git-filter-branch-keep-rewrites.

> @@ +41,5 @@
> > +    },
> > +    "backup_dir": "/mnt/netapp/github_sync/aki/%s" % hostname,
> > +    "cvs_manifest": CVS_MANIFEST,
> > +    "tooltool_servers": ["http://runtime-binaries.pvt.build.mozilla.org/tooltool/"],
> > +    "cvs_history_tarball": "/home/asasaki/mozilla-cvs-history.tar.bz2",
>
> Shouldn't the initial artifacts be in tooltool?

gd{2,3,4} can't reach tooltool, so this was deliberate.
I may change if the AWS boxes can, but then again I don't want to touch the initial conversion stuff too much now.

> @@ +96,5 @@
> > +            "branches": {
> > +                "default": "gecko-18",
> > +            },
> > +        },
> > +        "tag_config": {
>
> This tag_config is redundant, as both targets already have one specified.
>
> Not a functional issue, but it can leave humans confused about which wins :)

I can explain on a whiteboard maybe.
But basically: since the push is happening from a multi-repo conversion dir, and we don't save which tags were converted, a wildcard/regex tag_config for push can push tags from other repos (repo1 tag conversion is loose and gets uglytag1, prodtag1; repo2 tag conversion has a stricter conversion tag_config so it gets prodtag2 but skips uglytag2.  repo1 target tag push only pushes prodtag* so we get prodtag1 and prodtag2 downstream.  repo2 target pushes *, so we get uglytag1, prodtag1, prodtag2 downstream.  Because of this, I'm limiting on both ends for safety).

> ::: configs/vcs_sync/l10n.py
> @@ +1,5 @@
> > +from copy import deepcopy
> > +import os
> > +import socket
> > +hostname = socket.gethostname()
> > +
>
> I don't see any mapfile specifications here, but we do need one integrated
> mapfile for all l10n.
>
> How is that handled?

l10n is unfinished.  To be written.

> ::: configs/vcs_sync/project-branches.py
> @@ +7,5 @@
> > +PROJECT_BRANCHES = [
> > +    # twig projects
> > +    "ash",
> > +    "alder",
> > +    "ash",
>
> duplicate twig "ash"

Fixed.

> ::: mozharness/base/vcs/vcssync.py
> @@ +11,5 @@
> > +import smtplib
> > +import sys
> > +import time
> > +
> > +sys.path.insert(1, os.path.dirname(os.path.dirname(os.path.dirname(sys.path[0]))))
>
> Wow! that's cryptic. Is this a mozharnessism?

Yeah, basically adds ../../.. to the path to import python modules from.

> @@ +66,5 @@
> > +                "To: %s" % notify_config['to'],
> > +                "CC: %s" % ','.join(notify_config.get('cc', [])),
> > +                "Subject: %s" % subject,
> > +                "",
> > +                text
>
> SMTP lawyer here -- the body of the message is also supposed to use network
> format \r\n, but 'text' is built using only '\n'
>
> In this day and age, it'll probably work, but :)

It does work; I've been getting plenty of email.
(Assignee)

Comment 76

4 years ago
https://groups.google.com/d/msg/mozilla.dev.planning/hfZfm7vAHKA/fJNSQFlyUtIJ
Attachment #815555 - Flags: review?(rail) → review+
Comment on attachment 815574 [details] [diff] [review]
vcs_sync

Review of attachment 815574 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

::: scripts/vcs-sync/vcs_sync.py
@@ +271,4 @@
>          else:
> +            self.all_repos = list(self.config.get('conversion_repos', []))
> +        if self.config.get('conversion_type') == 'project-branches':
> +            self.all_repos += self._query_project_repos()

Does this mean that we have all project branches in git as branches? My only concern here is the fact that we reset project branches from time to time. Would this break conversion?
Attachment #815574 - Flags: review?(rail) → review+
(Assignee)

Comment 78

4 years ago
(In reply to Rail Aliiev [:rail] from comment #77)
> Comment on attachment 815574 [details] [diff] [review]
> vcs_sync
> 
> Review of attachment 815574 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> LGTM!
> 
> ::: scripts/vcs-sync/vcs_sync.py
> @@ +271,4 @@
> >          else:
> > +            self.all_repos = list(self.config.get('conversion_repos', []))
> > +        if self.config.get('conversion_type') == 'project-branches':
> > +            self.all_repos += self._query_project_repos()
> 
> Does this mean that we have all project branches in git as branches? My only
> concern here is the fact that we reset project branches from time to time.
> Would this break conversion?

Yes, https://github.com/mozilla/integration-gecko-projects/branches .
I tested with a user repo -- I landed a new non-fastforward head, then I reset the repo.  The non-fastforward head required manual fixing via |hg debugsetparents|.  The reset worked without human intervention, since the new head (based off m-c) was a fastforward from the old head.

If we get borked, we have the option of deleting the branch in github.  If we get completely borked, we have the option of deleting the entire integration-gecko-projects repo and starting over, as stated here: https://github.com/mozilla/integration-gecko-projects/blob/master/README.md
(Assignee)

Comment 79

4 years ago
Comment on attachment 815555 [details] [diff] [review]
script_cleanup

http://hg.mozilla.org/build/mozharness/rev/5b178e665fa0
Attachment #815555 - Flags: checked-in+
(Assignee)

Comment 80

4 years ago
Comment on attachment 815574 [details] [diff] [review]
vcs_sync

http://hg.mozilla.org/build/mozharness/rev/cd8ea7755bb4
Attachment #815574 - Flags: checked-in+
(Assignee)

Comment 81

4 years ago
Created attachment 816264 [details] [diff] [review]
latest changes

This patch gets us updated to the code I'm currently running on vcssync{1,2}.

* adds gitmo-beagle target to beagle.py
* updates github targets to official targets, rather than my user account for beagle + project-branches
* changes the upload locations to people.m.o, for now, for beagle + project-branches
* removes duplicate "ash" from the project list
* adds release+vcs2vcs to the notify list
* adds some copy_to_upload_dir() updates.  I'm now copying logs to the upload dir twice: once before the upload, and once at the end of the job (since there are still steps that can go wrong after the first copy)
* adds a --no-check-incoming option, which kind of forces an update-all and push-all

Asking :hwine since :rail's out til Wednesday.  Let me know if you have questions?

Also, I think I'll close this bug and reopen new enhancement bugs after this lands.
Attachment #816264 - Flags: review?(hwine)
something here is in mozharness production
Comment on attachment 816264 [details] [diff] [review]
latest changes

Review of attachment 816264 [details] [diff] [review]:
-----------------------------------------------------------------

The questions I have may have reasonable answers, so r+ assuming that is the case.

::: mozharness/base/script.py
@@ +1126,5 @@
>  
>              if not post_success:
>                  self.fatal("Aborting due to failure in post-run listener.")
> +        if self.config.get("copy_logs_post_run", True):
> +            self.copy_logs_to_upload_dir()

Just a question -- if we can't post, wouldn't we especially want the logs copied? Only true if the logs are useful for debugging that issue -- your call on how useful they would be.

::: scripts/vcs-sync/vcs_sync.py
@@ +781,5 @@
>          self._write_repo_update_json(repo_map)
>          if failure_msg:
>              self.fatal("Unable to push these repos:\n%s" % failure_msg)
>  
> +    def preflight_upload(self):

This is a new method, but I don't see it being invoked anywhere in this changeset.

If this was a previously un-implemented method on the base class, shouldn't it include a super() call for safety? If not, where is it being invoked?
Attachment #816264 - Flags: review?(hwine) → review+
(Assignee)

Comment 84

4 years ago
(In reply to Hal Wine [:hwine] (use needinfo) from comment #83)
> Comment on attachment 816264 [details] [diff] [review]
> latest changes
> 
> Review of attachment 816264 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> The questions I have may have reasonable answers, so r+ assuming that is the
> case.
> 
> ::: mozharness/base/script.py
> @@ +1126,5 @@
> >  
> >              if not post_success:
> >                  self.fatal("Aborting due to failure in post-run listener.")
> > +        if self.config.get("copy_logs_post_run", True):
> > +            self.copy_logs_to_upload_dir()
> 
> Just a question -- if we can't post, wouldn't we especially want the logs
> copied? Only true if the logs are useful for debugging that issue -- your
> call on how useful they would be.

This actually is outside the scope of post_success (see indentation :)
This is the base mozharness run() method, so we're copying logs/* to build/upload/logs/* every run, even if it doesn't make sense (e.g. in cases where we're not doing anything with build/upload/, which is most jobs these days).

This doesn't actually turn it off, but gives us a way to turn it off, if we set copy_logs_post_run to False.

> ::: scripts/vcs-sync/vcs_sync.py
> @@ +781,5 @@
> >          self._write_repo_update_json(repo_map)
> >          if failure_msg:
> >              self.fatal("Unable to push these repos:\n%s" % failure_msg)
> >  
> > +    def preflight_upload(self):
> 
> This is a new method, but I don't see it being invoked anywhere in this
> changeset.
> 
> If this was a previously un-implemented method on the base class, shouldn't
> it include a super() call for safety? If not, where is it being invoked?

We by default run preflight_ACTION() if it exists, then ACTION() (with dashes converted to underscores), then postflight_ACTION() for all enabled actions, per http://hg.mozilla.org/build/mozharness/file/7d9425c91051/mozharness/base/script.py#l1051 .
(Assignee)

Comment 85

4 years ago
Comment on attachment 816264 [details] [diff] [review]
latest changes

http://hg.mozilla.org/build/mozharness/rev/be4c6ce194dc
Attachment #816264 - Flags: checked-in+
(Assignee)

Comment 86

4 years ago
(In reply to Aki Sasaki [:aki] from comment #81)
> Also, I think I'll close this bug and reopen new enhancement bugs after this
> lands.

Yay!
Also https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED

Updated

3 years ago
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.