1116328 - can no longer push new repo to webheads

Status: VERIFIED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect, P1)

Description

[Hal Wine [:hwine] (use NI)]

invocation of /repo/hg/scripts/push-repo.sh fails silently (exit code '0')

Brief investigation shows issue with script on web heads:

[[e-mail]3 ha]# /usr/bin/ssh -l hg -i /etc/mercurial/mirror -o Server
AliveInterval=5 -o ConnectionAttempts=3 -o StrictHostKeyChecking=no -o ConnectTi
meout=10s -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o 
UserKnownHostsFile=/etc/mercurial/known_hosts hgweb10.dmz.scl3.mozilla.com -- ga
ia-l10n/ha
gaia-l10n/ha does not yet exist, cloning
no changes found
** Unknown exception encountered with possibly-broken third-party extension hgwe
bjson
** which supports versions unknown of Mercurial.
** Please disable hgwebjson and try your action again.
** If that fixes the bug please report it to the extension author.
** Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat
 4.4.7-4)]
** Mercurial Distributed SCM (version 3.2.3)
** Extensions loaded: hgwebjson, pushlog-feed, pushlog, buglink, serverlog
Traceback (most recent call last):
  File "/usr/bin/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 71, in d
ispatch
    ret = _runcatch(req)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 140, in _runcatch
    return _dispatch(req)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 850, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 611, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 941, in _runcommand
    return checkargs()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 912, in checkargs
    return cmdfunc()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 847, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 677, in check 
    return func(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 1371, in clone
    branch=opts.get('branch'))
  File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 423, in clone
    destpeer.local().clone(srcpeer, heads=revs, stream=stream)
  File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 1751, in clone
    ret = exchange.pull(self, remote, heads).cgresult
  File "/usr/lib64/python2.6/site-packages/mercurial/exchange.py", line 843, in pull
    pullop.closetransaction()
  File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 196, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/repo_local/mozilla/extensions/pushlog/__init__.py", line 90, in exchangepullpushlog
    raise Abort('error fetching pushlog: %s' % lines[1])
TypeError: 'listiterator' object is unsubscriptable
retrying hg clone --config hooks.pretxnchangegroup.z_linearhistory= --config hoo
ks.pretxnchangegroup.z_loghistory= --config trusted.users=root,hg --config paths
.default=ssh://hg.mozilla.org/gaia-l10n/ha -U -v ssh://hg.mozilla.org/gaia-l10n/
ha gaia-l10n/ha
no changes found
** Unknown exception encountered with possibly-broken third-party extension hgwe
bjson
** which supports versions unknown of Mercurial.
** Please disable hgwebjson and try your action again.
** If that fixes the bug please report it to the extension author.
** Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat
 4.4.7-4)]
** Mercurial Distributed SCM (version 3.2.3)
** Extensions loaded: hgwebjson, pushlog-feed, pushlog, buglink, serverlog
Traceback (most recent call last):
  File "/usr/bin/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 28, in r
un
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 71, in d
ispatch
    ret = _runcatch(req)   
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 140, in 
_runcatch
    return _dispatch(req)  
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 850, in 
_dispatch
    cmdpats, cmdoptions)   
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 611, in 
runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 941, in 
_runcommand
    return checkargs()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 912, in 
checkargs
    return cmdfunc()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 847, in 
<lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 677, in check   
    return func(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 1371, in clone
    branch=opts.get('branch'))
  File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 423, in clone
    destpeer.local().clone(srcpeer, heads=revs, stream=stream)
  File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 1751, in clone
    ret = exchange.pull(self, remote, heads).cgresult
  File "/usr/lib64/python2.6/site-packages/mercurial/exchange.py", line 843, in pull
    pullop.closetransaction()
  File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 196, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/repo_local/mozilla/extensions/pushlog/__init__.py", line 90, in exchangepullpushlog
    raise Abort('error fetching pushlog: %s' % lines[1])
TypeError: 'listiterator' object is unsubscriptable

Comment 1

[Gregory Szorc [:gps]]

A regression from the new pushlog extension!

Thanks for tracking down the stack trace.

Comment 2

[Gregory Szorc [:gps]]

There are 2 bugs here:

1) The exception is due to the error reporting code not working. Trying to access an iterator like a list. Derp. This is masking the real bug.

2) Setting a breakpoint, the error from the server is: "unable to open database file"

There is no pushlog2.db file in the gaia-l10n/ha repository. We *should* create the database/file on first access. Permissions look sane. Will have to set some breakpoints on the server to dig into this. This is my #1 priority today.

Comment 3

[Gregory Szorc [:gps]]

This smells like a permissions issue.

The .hg directory for gaia-l10n/ha on hgssh has the following permissions:

   drwxrwsr-x 3 hg   scm_l10n

If a su to 'hg', I can touch pushlog2.db just fine. If I run a standalone server as the 'hg' user, things also work. But when something similar is done via ssh, things don't work. I'm a bit confused.

Comment 4

[Gregory Szorc [:gps]]

And we have a culprit!

When 'hg' on hgweb ssh's into hgssh, the user gets turned into [e-mail]. And this user is not part of the scm_l10n group. So, it is unable to write the pushlog2.db file to the .hg directory since .hg is owned hg:scm_l10n.

If I su into a user that is part of the scm_l10n group, I can write the file just fine.

I /think/ the solution here is to ensure the [e-mail] is part of *all* the LDAP groups that own Mercurial repos.

Comment 5

[Hal Wine [:hwine] (use NI)]

the vcs-sync credentials were originally created for the use of vcs-sync, which ideally does not have access to any group. (I.e. it can only write to repos it is the owner of)

When ssh for webhead work was done, the credentials were appropriated for that as well, for expediency.

To unblock us, the credentials can be added to the needed groups, but the real solution is separate credentials for the web head pushes, and back to restricted permissions for vcs-sync

Comment 6

[Gregory Szorc [:gps]]

I'm working on patches to pushlog to work around this. We shouldn't need write permissions to return an empty set of pushlog entries if the pushlog file doesn't even exist.

Comment 7

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bz://1116328/gps

Comment 8

[Gregory Szorc [:gps]]

/r/1801 - pushlog: add test verifying empty/missing pushlog2.db clones
/r/1803 - pushlog: properly display server errors (bug 1116328)
/r/1805 - pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)

Pull down these commits:

hg pull review -r 08b68cd4bf30aca41a284243e406eca051fcb118

Comment 9

[Gregory Szorc [:gps]]

/r/1801 - pushlog: add test verifying empty/missing pushlog2.db clones
/r/1803 - pushlog: properly display server errors (bug 1116328)
/r/1805 - pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)

Pull down these commits:

hg pull review -r b374532e13e38c6fb623bcc27e52a015bde5ea6a

Comment 10

[Gregory Szorc [:gps]]

Per IRC discussion, let's hold off on making group membership changes and work around this in the pushlog extension. The linked patches address that.

Comment 11

[Steve Fink [:sfink] [:s:]]

https://reviewboard.mozilla.org/r/1805/#review1237

Looks right to me, though I must admit that as my use of reviewboard, I found it hard to understand which changes I am r+ing. The diff between the original and the current v2 looks good.

Comment 12

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/2771273acd0b
https://hg.mozilla.org/hgcustom/version-control-tools/rev/588abca63d93

Comment 13

[Gregory Szorc [:gps]]

(In reply to Steve Fink [:sfink, :s:] from comment #11)
> https://reviewboard.mozilla.org/r/1805/#review1237
> 
> Looks right to me, though I must admit that as my use of reviewboard, I
> found it hard to understand which changes I am r+ing. The diff between the
> original and the current v2 looks good.

Don't worry about it. The UI kind of sucks. It should be fixed by end of Q1.

Thanks for looking at this. I should have the new code deployed shortly.

Comment 14

[Gregory Szorc [:gps]]

This is now deployed. Confirmed that the command in comment #0 no longer results in clone error in production.

There are still some repos that don't exist on hgweb. These will need to be manually fixed.

Comment 15

[Hal Wine [:hwine] (use NI)]

(In reply to Gregory Szorc [:gps] from comment #14)
> There are still some repos that don't exist on hgweb. These will need to be
> manually fixed.

All fixed using normal repo setup process! Thanks for quick turn!

Comment 17

[Gregory Szorc [:gps]]

Attachment: MozReview Request: pushlog: add test verifying empty/missing pushlog2.db clones

Comment 18

[Gregory Szorc [:gps]]

Attachment: MozReview Request: pushlog: properly display server errors (bug 1116328)

Comment 19

[Gregory Szorc [:gps]]

Attachment: MozReview Request: pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)