657095 - mozilla-central pushlog broken, preventing any pushes

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phil Ringnalda (:philor)]

I wasn't around for the full story of what happened, but http://hg.mozilla.org/mozilla-central/rev/93604789da99 and the other things in that push were somehow an accidental push of a /projects/cedar/ tree to mozilla-central instead of cedar, and it seems to have broken mozilla-central. Pushing there now gives

searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 8 changes to 8 files
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: error: pretxnchangegroup.z_loghistory hook raised an exception: column rev is not unique
remote: transaction abort!
remote: rollback completed
remote: ** unknown exception encountered, details follow
remote: ** report bug details to http://mercurial.selenic.com/bts/
remote: ** or [e-mail]
remote: ** Python 2.4.3 (#1, Jun 11 2009, 14:09:58) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)]
remote: ** Mercurial Distributed SCM (version 1.5.4)
remote: ** Extensions loaded: hgwebjson, pushlog-feed, buglink
remote: Traceback (most recent call last):
remote:   File "/usr/bin/hg", line 27, in ?
remote:     mercurial.dispatch.run()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 16, in run
remote:     sys.exit(dispatch(sys.argv[1:]))
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 30, in dispatch
remote:     return _runcatch(u, args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 50, in _runcatch
remote:     return _dispatch(ui, args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 471, in _dispatch
remote:     return runcommand(lui, repo, cmd, fullargs, ui, options, d)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 341, in runcommand
remote:     ret = _runcommand(ui, options, cmd, d)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 522, in _runcommand
remote:     return checkargs()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 476, in checkargs
remote:     return cmdfunc()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 470, in <lambda>
remote:     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/util.py", line 401, in check
remote:     return func(*args, **kwargs)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/commands.py", line 2904, in serve
remote:     s.serve_forever()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 45, in serve_forever
remote:     while self.serve_one():
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 57, in serve_one
remote:     impl()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 208, in do_unbundle
remote:     r = self.repo.addchangegroup(fp, 'serve', self.client_url())
remote:   File "/usr/lib/python2.4/site-packages/mercurial/localrepo.py", line 2120, in addchangegroup
remote:     url=url, pending=p)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/localrepo.py", line 152, in hook
remote:     return hook.hook(self.ui, self, name, throw, **args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/hook.py", line 142, in hook
remote:     r = _pythonhook(ui, repo, name, hname, hookfn, args, throw) or r
remote:   File "/usr/lib/python2.4/site-packages/mercurial/hook.py", line 68, in _pythonhook
remote:     r = obj(ui=ui, repo=repo, hooktype=name, **args)
remote:   File "/usr/lib/python2.4/site-packages/mozhghooks/pushlog.py", line 79, in log
remote:     (pushid, ctx.rev(), hex(ctx.node())))
remote: pysqlite2.dbapi2.IntegrityError: column rev is not unique
abort: unexpected response: empty string

mozilla-central's closed, thus the severity, though closing it is sort of pointless since nobody can push to it anyway.

Comment 1

[Justin Dow [:jabba]]

I paged noahm for his help, but I'll see if there is anything I can do in the meantime.

Comment 2

[Noah Meyerhans [:noahm]]

OK, I've cleaned up the bogus entry in pushlog.  Change 69510:93604789da99 is still there, but you guys should be able to back it out now.

Comment 3

[Chris AtLee [:catlee]]

What was the bogus entry in the pushlog?

Comment 4

[Justin Dow [:jabba]]

[22:02]  <noahm> [21:17:08] ok, so it looks like that's what's wrong...
[22:02]  <noahm> [21:17:19] there is an entry in pushlog that is not in the repo itself
[22:02]  <noahm> [21:17:41] notice that the most recent changeset in hg is 69510
[22:02]  <noahm> [21:17:52] but there is a 69511 according to pushlog

I didn't save the output from last night when we were looking at it, but changeset 69511 is what noahm removed from pushlog, since it was a bogus entry (didn't match anything in the repo).

Comment 5

[Jason Duell]

> were somehow an accidental push of a /projects/cedar/ tree to 
> mozilla-central instead of cedar

Yes, it was a mindless cut-and-paste of my 'default-push' setting from my m-c hgrc file to my cedar one that resulted in me pushing the cedar head to m-c.  I was going to back the whole mess out, but various people on IRC who Seemed to Know suggested it actually ought to be fine to leave it.  Sorry for the mess and extra work.