can no longer push new repo to webheads

VERIFIED FIXED

Status

Developer Services
Mercurial: hg.mozilla.org
P1
normal
VERIFIED FIXED
3 years ago
2 years ago

People

(Reporter: hwine, Assigned: gps)

Tracking

({regression})

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4251] )

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

3 years ago
invocation of /repo/hg/scripts/push-repo.sh fails silently (exit code '0')

Brief investigation shows issue with script on web heads:

[root@hgssh1.dmz.scl3 ha]# /usr/bin/ssh -l hg -i /etc/mercurial/mirror -o Server
AliveInterval=5 -o ConnectionAttempts=3 -o StrictHostKeyChecking=no -o ConnectTi
meout=10s -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o 
UserKnownHostsFile=/etc/mercurial/known_hosts hgweb10.dmz.scl3.mozilla.com -- ga
ia-l10n/ha
gaia-l10n/ha does not yet exist, cloning
no changes found
** Unknown exception encountered with possibly-broken third-party extension hgwe
bjson
** which supports versions unknown of Mercurial.
** Please disable hgwebjson and try your action again.
** If that fixes the bug please report it to the extension author.
** Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat
 4.4.7-4)]
** Mercurial Distributed SCM (version 3.2.3)
** Extensions loaded: hgwebjson, pushlog-feed, pushlog, buglink, serverlog
Traceback (most recent call last):
  File "/usr/bin/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 71, in d
ispatch
    ret = _runcatch(req)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 140, in _runcatch
    return _dispatch(req)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 850, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 611, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 941, in _runcommand
    return checkargs()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 912, in checkargs
    return cmdfunc()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 847, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 677, in check 
    return func(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 1371, in clone
    branch=opts.get('branch'))
  File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 423, in clone
    destpeer.local().clone(srcpeer, heads=revs, stream=stream)
  File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 1751, in clone
    ret = exchange.pull(self, remote, heads).cgresult
  File "/usr/lib64/python2.6/site-packages/mercurial/exchange.py", line 843, in pull
    pullop.closetransaction()
  File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 196, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/repo_local/mozilla/extensions/pushlog/__init__.py", line 90, in exchangepullpushlog
    raise Abort('error fetching pushlog: %s' % lines[1])
TypeError: 'listiterator' object is unsubscriptable
retrying hg clone --config hooks.pretxnchangegroup.z_linearhistory= --config hoo
ks.pretxnchangegroup.z_loghistory= --config trusted.users=root,hg --config paths
.default=ssh://hg.mozilla.org/gaia-l10n/ha -U -v ssh://hg.mozilla.org/gaia-l10n/
ha gaia-l10n/ha
no changes found
** Unknown exception encountered with possibly-broken third-party extension hgwe
bjson
** which supports versions unknown of Mercurial.
** Please disable hgwebjson and try your action again.
** If that fixes the bug please report it to the extension author.
** Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat
 4.4.7-4)]
** Mercurial Distributed SCM (version 3.2.3)
** Extensions loaded: hgwebjson, pushlog-feed, pushlog, buglink, serverlog
Traceback (most recent call last):
  File "/usr/bin/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 28, in r
un
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 71, in d
ispatch
    ret = _runcatch(req)   
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 140, in 
_runcatch
    return _dispatch(req)  
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 850, in 
_dispatch
    cmdpats, cmdoptions)   
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 611, in 
runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 941, in 
_runcommand
    return checkargs()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 912, in 
checkargs
    return cmdfunc()
  File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 847, in 
<lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 677, in check   
    return func(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 1371, in clone
    branch=opts.get('branch'))
  File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 423, in clone
    destpeer.local().clone(srcpeer, heads=revs, stream=stream)
  File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 1751, in clone
    ret = exchange.pull(self, remote, heads).cgresult
  File "/usr/lib64/python2.6/site-packages/mercurial/exchange.py", line 843, in pull
    pullop.closetransaction()
  File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 196, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/repo_local/mozilla/extensions/pushlog/__init__.py", line 90, in exchangepullpushlog
    raise Abort('error fetching pushlog: %s' % lines[1])
TypeError: 'listiterator' object is unsubscriptable
A regression from the new pushlog extension!

Thanks for tracking down the stack trace.
Depends on: 1065771
Keywords: regression

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4251]
There are 2 bugs here:

1) The exception is due to the error reporting code not working. Trying to access an iterator like a list. Derp. This is masking the real bug.

2) Setting a breakpoint, the error from the server is: "unable to open database file"

There is no pushlog2.db file in the gaia-l10n/ha repository. We *should* create the database/file on first access. Permissions look sane. Will have to set some breakpoints on the server to dig into this. This is my #1 priority today.
Assignee: nobody → gps
Status: NEW → ASSIGNED
This smells like a permissions issue.

The .hg directory for gaia-l10n/ha on hgssh has the following permissions:

   drwxrwsr-x 3 hg   scm_l10n

If a su to 'hg', I can touch pushlog2.db just fine. If I run a standalone server as the 'hg' user, things also work. But when something similar is done via ssh, things don't work. I'm a bit confused.
And we have a culprit!

When 'hg' on hgweb ssh's into hgssh, the user gets turned into vcs-sync@mozilla.com. And this user is not part of the scm_l10n group. So, it is unable to write the pushlog2.db file to the .hg directory since .hg is owned hg:scm_l10n.

If I su into a user that is part of the scm_l10n group, I can write the file just fine.

I /think/ the solution here is to ensure the vcs-sync@mozilla.com is part of *all* the LDAP groups that own Mercurial repos.
(Reporter)

Comment 5

3 years ago
the vcs-sync credentials were originally created for the use of vcs-sync, which ideally does not have access to any group. (I.e. it can only write to repos it is the owner of)

When ssh for webhead work was done, the credentials were appropriated for that as well, for expediency.

To unblock us, the credentials can be added to the needed groups, but the real solution is separate credentials for the web head pushes, and back to restricted permissions for vcs-sync
I'm working on patches to pushlog to work around this. We shouldn't need write permissions to return an empty set of pushlog entries if the pushlog file doesn't even exist.
Created attachment 8542627 [details]
MozReview Request: bz://1116328/gps
Attachment #8542627 - Flags: review?(sphink)
/r/1801 - pushlog: add test verifying empty/missing pushlog2.db clones
/r/1803 - pushlog: properly display server errors (bug 1116328)
/r/1805 - pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)

Pull down these commits:

hg pull review -r 08b68cd4bf30aca41a284243e406eca051fcb118
/r/1801 - pushlog: add test verifying empty/missing pushlog2.db clones
/r/1803 - pushlog: properly display server errors (bug 1116328)
/r/1805 - pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)

Pull down these commits:

hg pull review -r b374532e13e38c6fb623bcc27e52a015bde5ea6a
Per IRC discussion, let's hold off on making group membership changes and work around this in the pushlog extension. The linked patches address that.
Priority: -- → P1
https://reviewboard.mozilla.org/r/1805/#review1237

Looks right to me, though I must admit that as my use of reviewboard, I found it hard to understand which changes I am r+ing. The diff between the original and the current v2 looks good.
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2771273acd0b
https://hg.mozilla.org/hgcustom/version-control-tools/rev/588abca63d93
(In reply to Steve Fink [:sfink, :s:] from comment #11)
> https://reviewboard.mozilla.org/r/1805/#review1237
> 
> Looks right to me, though I must admit that as my use of reviewboard, I
> found it hard to understand which changes I am r+ing. The diff between the
> original and the current v2 looks good.

Don't worry about it. The UI kind of sucks. It should be fixed by end of Q1.

Thanks for looking at this. I should have the new code deployed shortly.
This is now deployed. Confirmed that the command in comment #0 no longer results in clone error in production.

There are still some repos that don't exist on hgweb. These will need to be manually fixed.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 15

3 years ago
(In reply to Gregory Szorc [:gps] from comment #14)
> There are still some repos that don't exist on hgweb. These will need to be
> manually fixed.

All fixed using normal repo setup process! Thanks for quick turn!
Status: RESOLVED → VERIFIED
Attachment #8542627 - Flags: review?(sphink) → review+

Updated

3 years ago
Blocks: 1116899
Comment on attachment 8542627 [details]
MozReview Request: bz://1116328/gps
Attachment #8542627 - Attachment is obsolete: true
Attachment #8618995 - Flags: review+
Attachment #8618996 - Flags: review+
Attachment #8618997 - Flags: review+
Created attachment 8618995 [details]
MozReview Request: pushlog: add test verifying empty/missing pushlog2.db clones
Created attachment 8618996 [details]
MozReview Request: pushlog: properly display server errors (bug 1116328)
Created attachment 8618997 [details]
MozReview Request: pushlog: do not attempt to create pushlog for read-only operation (bug 1116328)
You need to log in before you can comment on or make changes to this bug.