835202 - Exception exporting mozilla-central to github

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

From 'alert_major_errors alert' email:

Date: Sun, 27 Jan 2013 18:00:06 -0800
To: [e-mail]
Subject: [vcs2vcs] alert_major_errors alert
From: [e-mail]

/opt/vcs2vcs/logs/job02/update.log:2013-01-27T16:16:14-0800;6:48.90;408.90;49.29;358.66;1572016;896;168;152;127272;0;0;1;hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central gexport;
/opt/vcs2vcs/logs/job02/update.log:2013-01-27T16:33:17-0800;0:10.90;10.90;0.36;10.51;1318480;26;9;0;124856;0;0;1;hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central gexport;
/opt/vcs2vcs/logs/job02/update.log:2013-01-27T16:43:42-0800;0:10.79;10.79;0.34;10.42;1318512;23;5;0;124856;0;0;1;hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central gexport;
/opt/vcs2vcs/logs/job02/update.log:2013-01-27T16:54:06-0800;0:10.76;10.76;0.31;10.42;1318496;22;7;0;124856;0;0;1;hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central gexport;

ie process has exit status of 1

From logs/job02/details.log:

2013-01-27T16:15:59-0800: starting: hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central pull
pulling from http://hg.mozilla.org/mozilla-central
searching for changes
adding changesets
adding manifests
adding file changes
added 38 changesets with 123 changes to 98 files
(run 'hg update' to get a working copy)
2013-01-27T16:16:03-0800: finished: hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central pull
2013-01-27T16:16:03-0800: starting: hg --cwd /opt/vcs2vcs/repos/releases-mozilla-central bookmark -f -r default master
....
exporting hg objects to git
** unknown exception encountered, please report by visiting
**  http://mercurial.selenic.com/wiki/BugTracker
** Python 2.6.6 (r266:84292, Sep 12 2011, 14:03:14) [GCC 4.4.5 20110214 (Red Hat 4.4.5-6)]
** Mercurial Distributed SCM (version 2.2.1)
** Extensions loaded: hggit
Traceback (most recent call last):
  File "/opt/vcs2vcs/bin/hg", line 5, in <module>
    pkg_resources.run_script('mercurial==2.2.1', 'hg')
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 489, in run_script
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 1207, in run_script
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/EGG-INFO/scripts/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 27, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 64, in dispatch
    return _runcatch(req)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 87, in _runcatch
    return _dispatch(req)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 685, in _dispatch
    cmdpats, cmdoptions)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 467, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 775, in _runcommand
    return checkargs()
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 746, in checkargs
    return cmdfunc()
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/dispatch.py", line 682, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/opt/vcs2vcs/venv/hg/lib/python2.6/site-packages/mercurial-2.2.1-py2.6-linux-x86_64.egg/mercurial/util.py", line 463, in check
    return func(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/hggit/__init__.py", line 101, in gexport
  File "build/bdist.linux-x86_64/egg/hggit/git_handler.py", line 193, in export_commits
  File "build/bdist.linux-x86_64/egg/hggit/git_handler.py", line 290, in export_git_objects
  File "build/bdist.linux-x86_64/egg/hggit/git_handler.py", line 319, in export_hg_commit
ValueError: invalid literal for int() with base 10: '<[e-mail]>'

The new changes were:
 http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=aafdf3566582&tochange=63685729b771
and 
 http://hg.mozilla.org/mozilla-central/rev/3bc89c4fedd7
looks mighty suspicious.

Comment 1

[u429623]

this job disabled while the bug is sorted out and addressed. (job02 on gd0)

Comment 2

[u429623]

additional info - original commit made by bsmith using -u switch with hg windows client 2.4.2

recent change was to start using "committer extension" from https://bitbucket.org/ede/committer

Comment 3

[u429623]

Attachment: hotfix to hggit

This patch has been tested on a copy of the production repo, and produces the following output for the problem commit and it's child:

    (hwine)[[e-mail]3 releases-mozilla-central]$ git log --pretty=fuller -r v1.0.0 -n 2                                                               
    commit a0fa58117e903f26ab0eda88a84827969aee91c6
    Author:     Shane Caraveo <[e-mail]>
    AuthorDate: Sat Jan 26 16:48:54 2013 -0800
    Commit:     Shane Caraveo <[e-mail]>
    CommitDate: Sat Jan 26 16:48:54 2013 -0800

        Bug 782850 fix nsContextMenu to work in social content panels, r=felipe

    commit 1bb99306b32db6c15cb942532fa77b522d127523
    Author:     Carmen Jiménez Cabezas <[e-mail]>
    AuthorDate: Fri Jan 25 15:36:36 2013 -0800
    Commit:     Brian Smith <[e-mail]>
    CommitDate: Fri Jan 25 23:36:36 2013 +0000

        Bug 834091: Verify certificate chain for signed B2G apps as of the current time (now) instead of the signing time, r=bsmith
        
        --HG--
        extra : amend_source : 86d8ca2b28259aaf41983740b809ef8a51befc4f
        extra : rebase_source : e5a1c1199756e929f14852f5c83ba28d097449f4
    (hwine)[[e-mail]3 releases-mozilla-central]$

Comment 4

[(no longer active)]

I think this patch is not correct, since it will continue to do the handling specific to hg-git converted hg changesets with regard to the extra[committer] field even if that field is not coming from that, which is the case with the changeset in question.

I believe <https://groups.google.com/forum/?fromgroups=#!topic/hg-git/UE65Ba-FXiE> is a better approach.  I've already deployed that patch to the github mirror, FWIW, and it's been tested in the past few days.

Comment 5

[u429623]

Attachment: hotfix to hggit v 0.3.2

The functional difference between this and Ehsan's cited in comment 4 is how the committer date is displayed. (my original yielded UTC, Ehsan's yields local time)

Not only is that a more user friendly display, since Ehsan's patch is likely to get accepted upstream, changing to use his date selection. Both patches now yield identical sha1 sums for converted commits. (This will simplify incorporating later hggit patches.)

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 707359 [details] [diff] [review]
hotfix to hggit v 0.3.2

Review of attachment 707359 [details] [diff] [review]:
-----------------------------------------------------------------

r+ because this is closest to what was submitted upstream, and not worth bikeshedding over. I'm not a fan of the amount of lines changed though, but either way want this underlying issue fixed to get replication going again.

Also it works, thats always a plus.

Comment 7

[u429623]

packaged hg_git-0.3.2-moz1.tar.gz and placed on hg_git-0.3.2-moz1.tar.gz in production/python-packages

Comment 8

[u429623]

deployed to all production servers, repos have caught up for all b2g work.

Comment 9

[Aki Sasaki (not active)]

I hit this bug when converting m-c using the latest upstream, which concerns me.

Comment 10

[(no longer active)]

(In reply to comment #9)
> I hit this bug when converting m-c using the latest upstream, which concerns
> me.

Hmm, Hal upstreamed this fix: <https://groups.google.com/forum/?fromgroups=#!topic/hg-git/-M0xSybPGK8> so that's really strange.  Are you using a late enough version of hg-git?

Comment 11

[Aki Sasaki (not active)]

Attachment: corresponding log

It's barfing on the exact same revision.
I'm using hg-git revision 9edc7f19e6387c7c2ca20447510201b0a56e071c .
I'm going to guess Ehsan's upstream fix doesn't completely deal with this error.

Comment 12

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> (In reply to comment #9)
> > I hit this bug when converting m-c using the latest upstream, which concerns
> > me.
> 
> Hmm, Hal upstreamed this fix:
> <https://groups.google.com/forum/?fromgroups=#!topic/hg-git/-M0xSybPGK8> so
> that's really strange.  Are you using a late enough version of hg-git?

I think the main difference here is Hal's fork has a try/except:

https://hg.mozilla.org/users/hwine_mozilla.com/hg-git/file/4e2375e0d49f/hggit/git_handler.py#l315

Comment 13

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #12)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> > (In reply to comment #9)
> > > I hit this bug when converting m-c using the latest upstream, which concerns
> > > me.
> > 
> > Hmm, Hal upstreamed this fix:
> > <https://groups.google.com/forum/?fromgroups=#!topic/hg-git/-M0xSybPGK8> so
> > that's really strange.  Are you using a late enough version of hg-git?

According to comment 4, you upstreamed https://groups.google.com/forum/?fromgroups=#!topic/hg-git/UE65Ba-FXiE for this.

> I think the main difference here is Hal's fork has a try/except:
> 
> https://hg.mozilla.org/users/hwine_mozilla.com/hg-git/file/4e2375e0d49f/
> hggit/git_handler.py#l315

I edited the git_handler.py in my venv site-packages to include the try/except, and I'm past this error.

We'll either need to upstream this, or we'll be stuck using a fork.

Comment 14

[(no longer active)]

(In reply to Aki Sasaki [:aki] from comment #13)
> (In reply to Aki Sasaki [:aki] from comment #12)
> > (In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> > > (In reply to comment #9)
> > > > I hit this bug when converting m-c using the latest upstream, which concerns
> > > > me.
> > > 
> > > Hmm, Hal upstreamed this fix:
> > > <https://groups.google.com/forum/?fromgroups=#!topic/hg-git/-M0xSybPGK8> so
> > > that's really strange.  Are you using a late enough version of hg-git?
> 
> According to comment 4, you upstreamed
> https://groups.google.com/forum/?fromgroups=#!topic/hg-git/UE65Ba-FXiE for
> this.

Oh I see.  Well, I actually never finished upstreaming that patch, as they requested that I write a unit test for it, and I never got enough time to do that, so that sort of got stalled.  :(

(In my local version of hg-git, I do have that patch applied.)

> > I think the main difference here is Hal's fork has a try/except:
> > 
> > https://hg.mozilla.org/users/hwine_mozilla.com/hg-git/file/4e2375e0d49f/
> > hggit/git_handler.py#l315
> 
> I edited the git_handler.py in my venv site-packages to include the
> try/except, and I'm past this error.
> 
> We'll either need to upstream this, or we'll be stuck using a fork.

I still think that the patch I posted to the hg-git mailing list is better than the exception handling approach, as the latter might mask other exceptions that we did not intend to handle here.

But I do agree that we need to upstream this.  Do you have some time to write a unit test for this patch by any chance?  It should be fairly easy to do that by just using the committer extension...

Comment 15

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #14)
> > We'll either need to upstream this, or we'll be stuck using a fork.
> 
> I still think that the patch I posted to the hg-git mailing list is better
> than the exception handling approach, as the latter might mask other
> exceptions that we did not intend to handle here.

Ok.

> But I do agree that we need to upstream this.  Do you have some time to
> write a unit test for this patch by any chance?  It should be fairly easy to
> do that by just using the committer extension...

Using 0.4.0+ upstream resulted in different git shas.
We're investigating further.
It may be that we're stuck on a previous hg-git fork due to this.

If we can make upstream work, I can potentially look into the unit test.
Otherwise, it seems somewhat tangential to our immediate needs.

Comment 16

[(no longer active)]

(In reply to comment #15)
> > But I do agree that we need to upstream this.  Do you have some time to
> > write a unit test for this patch by any chance?  It should be fairly easy to
> > do that by just using the committer extension...
> 
> Using 0.4.0+ upstream resulted in different git shas.
> We're investigating further.

Sadface. :(  Thanks for looking into this.

> It may be that we're stuck on a previous hg-git fork due to this.
> 
> If we can make upstream work, I can potentially look into the unit test.
> Otherwise, it seems somewhat tangential to our immediate needs.

Hmm, I've never seen this kind of thing with previous hg-git versions...  I actually can't think of a good reason why there should be an intentional change in the way that we generate commit sha1's.  (Hopefully this is only a change in the commit sha1's and not on blobs/trees, right?)  If this is a regression, it would be fantastic if you could find what caused it so that we can report and fix it as soon as possible...

Comment 17

[Aki Sasaki (not active)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #16)
> Hmm, I've never seen this kind of thing with previous hg-git versions...  I
> actually can't think of a good reason why there should be an intentional
> change in the way that we generate commit sha1's.  (Hopefully this is only a
> change in the commit sha1's and not on blobs/trees, right?)  If this is a
> regression, it would be fantastic if you could find what caused it so that
> we can report and fix it as soon as possible...

I'm guessing https://github.com/escapewindow/hg-git/commit/f480d062057645241dce21e05c5aa85bc047b9dd will remove this sha discrepancy, and I'm hoping there aren't any more.  I don't think it's a regression.

Porting gps' patch to Hal's fork was taking a lot more uplifting, but is also an option if we keep hitting sha differences.

Comment 18

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #17)
> I'm guessing
> https://github.com/escapewindow/hg-git/commit/
> f480d062057645241dce21e05c5aa85bc047b9dd will remove this sha discrepancy,
> and I'm hoping there aren't any more.  I don't think it's a regression.

Using my github fork, I was able to convert m-c without changing shas.