1589382 - mozilla-unified not updating

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[James Graham [:jgraham]]

It looks like the bookmarks for mozilla unified haven't updated for some time (the latest revision pointed at by a bookmark is from 10 hours ago). I've noticed slowness recently (taking 30 minutes to update bookmarks) but this is worse.

Comment 1

[Connor Sheehan [:sheehan]]

Looks like c3bfb7519ae9 on autoland doesn't have any pushlog information and the unify process bails out when it finds that changeset.

Comment 2

[Connor Sheehan [:sheehan]]

I found this in the blackbox logs for autoland on hgssh1:

Traceback (most recent call last):
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 384, in _callcatch
    return scmutil.callcatch(ui, func)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/scmutil.py", line 167, in callcatch
    return func()
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 367, in _runcatchfunc
    return _dispatch(req)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 1021, in _dispatch
    cmdpats, cmdoptions)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 756, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 1030, in _runcommand
    return cmdfunc()
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/dispatch.py", line 1018, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/util.py", line 1680, in check
    return func(*args, **kwargs)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/commands.py", line 5297, in serve
    s.serve_forever()
  File "/var/hg/version-control-tools/hgext/serverlog/__init__.py", line 488, in serve_forever
    return super(sshserverwrapped, self).serve_forever()
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/wireprotoserver.py", line 796, in serve_forever
    self.serveuntil(threading.Event())
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/wireprotoserver.py", line 802, in serveuntil
    _runsshserver(self._ui, self._repo, self._fin, self._fout, ev)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/wireprotoserver.py", line 657, in _runsshserver
    rsp = wireprotov1server.dispatch(repo, proto, request)
  File "/var/hg/version-control-tools/pylib/vcsreplicator/vcsreplicator/hgext.py", line 546, in wireprotodispatch
    return orig(repo, proto, command)
  File "/var/hg/version-control-tools/hgext/serverlog/__init__.py", line 281, in wrappeddispatch
    res = orig(repo, proto, command)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/wireprotov1server.py", line 76, in dispatch
    return func(repo, proto, *args)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/wireprotov1server.py", line 597, in unbundle
    proto.client())
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/exchange.py", line 2423, in unbundle
    lockandtr[2].close()
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/transaction.py", line 47, in _active
    return func(self, *args, **kwds)
  File "/var/hg/venv_pash/lib/python2.7/site-packages/mercurial/transaction.py", line 529, in close
    self._postclosecallback[cat](self)
  File "/var/hg/version-control-tools/hgext/pushlog/__init__.py", line 215, in pushlog_tr_post_close
    conn.commit()
OperationalError: database is locked

We should definitely retry there. Still coming up with a solution to get unified back on track.

Comment 3

[Connor Sheehan [:sheehan]]

We fixed this by manually inserting an entry in the pushlog for the bad changeset. For archaeology sake:

  $ sudo sqlite3 /repo/hg/mozilla/integration/autoland/.hg/pushlog2.db
  SQLite version 3.7.17 2013-05-20 00:56:22
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> insert into pushlog (user, date) values ('[e-mail]', 1571270021);
  sqlite> SELECT id from pushlog ORDER BY id DESC limit 1;
  98230
  sqlite> insert into changesets (pushid, rev, node) values (98230, 497957, 'c3bfb7519ae9c9ef4198e0fdfc6b8d399414ef2d');
  sqlite> .exit
  $ /var/hg/venv_tools/bin/hg -R /repo/hg/mozilla/integration/autoland replicatesync