551240 - Unable to push to mozilla-central: pysqlite2.dbapi2.OperationalError: database is locked

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Benjamin Smedberg]

bz and I both tried to push to mozilla-central and have been receiving this error consistently over the past 30 minutes:

$ hg push
pushing to ssh://hg.mozilla.org/mozilla-central
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 2 changesets with 10 changes to 10 files
remote: error: pretxnchangegroup.z_linearhistory hook raised an exception: datab
ase is locked
remote: transaction abort!
remote: rollback completed
remote: ** unknown exception encountered, details follow
remote: ** report bug details to http://www.selenic.com/mercurial/bts
remote: ** or [e-mail]
remote: ** Mercurial Distributed SCM (version 1.2.1)
remote: ** Extensions loaded: buglink, hgwebjson, pushlog-feed
remote: Traceback (most recent call last):
remote:   File "/usr/bin/hg", line 27, in ?
remote:     mercurial.dispatch.run()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 16
, in run
remote:     sys.exit(dispatch(sys.argv[1:]))
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 25
, in dispatch
remote:     return _runcatch(u, args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 41
, in _runcatch
remote:     return _dispatch(ui, args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 37
2, in _dispatch
remote:     return runcommand(lui, repo, cmd, fullargs, ui, options, d)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 24
7, in runcommand
remote:     ret = _runcommand(ui, options, cmd, d)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 41
7, in _runcommand
remote:     return checkargs()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 37
7, in checkargs
remote:     return cmdfunc()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/dispatch.py", line 37
1, in <lambda>
remote:     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/util.py", line 718, i
n check
remote:     return func(*args, **kwargs)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/commands.py", line 26
41, in serve
remote:     s.serve_forever()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 4
0, in serve_forever
remote:     while self.serve_one(): pass
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 4
7, in serve_one
remote:     if impl: impl()
remote:   File "/usr/lib/python2.4/site-packages/mercurial/sshserver.py", line 1
94, in do_unbundle
remote:     r = self.repo.addchangegroup(fp, 'serve', self.client_url())
remote:   File "/usr/lib/python2.4/site-packages/mercurial/localrepo.py", line 2
061, in addchangegroup
remote:     url=url, pending=p)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/localrepo.py", line 1
26, in hook
remote:     return hook.hook(self.ui, self, name, throw, **args)
remote:   File "/usr/lib/python2.4/site-packages/mercurial/hook.py", line 113, i
n hook
remote:     args, throw) or r
remote:   File "/usr/lib/python2.4/site-packages/mercurial/hook.py", line 51, in
 _pythonhook
remote:     r = obj(ui=ui, repo=repo, hooktype=name, **args)
remote:   File "/usr/lib/python2.4/site-packages/mozhghooks/pushlog.py", line 57
, in log
remote:     conn.commit()
remote: pysqlite2.dbapi2.OperationalError: database is locked
abort: unexpected response: empty string

Comment 1

[(not currently active) Ted Mielczarek]

I'm also failing to load the pushlog webpage:
http://hg.mozilla.org/mozilla-central/pushloghtml

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I was getting this yesterday as well.  Other people were able to push while I was getting the error.

Comment 4

[Aravind Gottipati [:aravind]]

working on it, rebooting the backend servers.  They were pretty much dead.  The site should be coming back up now.  I am working on adding more backend servers to handle the traffic.

Comment 5

[Philipp Kewisch [:Fallen][☁️📆]]

Possibly related, when pulling from comm-central I get:

abort: HTTP Error 502: Bad Gateway

Comment 6

[Aravind Gottipati [:aravind]]

Okay, servers should be up now and working.  Please comment here if you are still having problems committing.

Comment 7

[Aravind Gottipati [:aravind]]

This should be fixed now.  If you are interested, 550280 is tracking the "add more backend servers" effort.