437637 - update pushlog hook/extension on hg.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[(not currently active) Ted Mielczarek]

I've modified bsmedberg's pushlog hook and extension to use a sqlite backend. We should update the copy on hg.mozilla.org to use this.

You're going to need to install the Python sqlite3 module to get this all to work. Hopefully that's not a big issue, I tested this all on OS X and it works fine, but I don't have a CentOS machine handy to test on.

The new hook is an in-process Python module, so you'll need to install it system-wide first, just do:
python setup.py install

The hgrc will need to be updated as well, the example-hgrc in the hghooks repo has been updated accordingly:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hghooks/file/229cd704c10e/example-hgrc

There's a python script included to convert the existing flat-file into a database:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hghooks/file/229cd704c10e/convert-pushlog-db.py

Just run it like:
python convert-pushlog-db mozilla-central/.hg/pushlog

The pushglog-feed update is just a straight update, there should be no configuration changes required, simply update the Python file to the new version:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hgpoller/file/904ea0ddd0b1/pushlog-feed.py

Comment 1

[Benjamin Smedberg]

Additional information: currently the pretxnchangegroup.z_linearhistory hook is only enabled for selected repos (I think mozilla-central and a few others). It should now be enabled for all repos by putting the setting in /etc/mercurial/hgrc

The pretxnchangegroup.a_singlehead hook is still only for selected repositories.

You can check to see if the python sqlite3 module is installed by doing
$ python
>>> import sqlite3

Comment 2

[Axel Hecht [:Pike]]

I was wondering if we should have only one pushlog for all the l10n reps. I'm not so sure that polling the pushlog feed for several dozen reps every few minutes is going to be fun for either the buildmaster nor the hg server. Should we really enable the pushlog for all reps before we have an idea how to fix that?

Comment 3

[Benjamin Smedberg]

Axel, that needs to be a different bug. I don't think it will be a problem.

Comment 4

[Aravind Gottipati [:aravind]]

I am ready to start working on this bug, but I have some concerns.

1.  Why sqllite?  plain text isn't good enough?  Adding more library dependencies will only slow things down.

2.  Also, w.r.t how this python module is implemented, Why can't I get a plain source.py that I drop in somewhere to use it?  Do I need the easy_install mess on the server just to install this module?

3. Why are we forcing ALL repos to have linear history?

Comment 5

[Benjamin Smedberg]

1) sqlite because we need the ability to query backwards and forwards in history, and the plaintext files will quickly become difficult to query efficiently

2) The Mercurial system does "import mozhghooks.pushlog"... you don't have to do the easy_install method, you could just copy the files manually to /usr/lib/python2.x/site-packages/mozhghooks, but easy_install does it for you

3) we're turning on the pushlog for all repositories. We are *not* turning on the linear-history hook for all repositories.

Comment 6

[Aravind Gottipati [:aravind]]

Can we use sqlite2 for this stuff?  sqlite3 works is new in python 2.5 (according to http://docs.python.org/lib/module-sqlite3.html and we have python 2.4. (we are already at the latest and greatest rhel release).

Would make my life a lot simpler if you can working with sqlite2.

Comment 7

[(not currently active) Ted Mielczarek]

That should work, I'll need to tweak the code slightly to cope. Let me test it tomorrow and if it works I'll push my changes.

Comment 8

[(not currently active) Ted Mielczarek]

I've adapted the Python code in all places to fall back to pysqlite2 if necessary:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hghooks/rev/6bbc8faf8b09
http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hgpoller/rev/3e98a662394d

You should be good to go.

Comment 9

[Aravind Gottipati [:aravind]]

Should be working now.

All repos now use this.

[hooks]
pretxnchangegroup.z_linearhistory = python:mozhghooks.pushlog.log


And I updated pushlog.py from the revisions listed above.

I also converted all exisiting pushlogs to sqlite db format.

Comment 10

[Aravind Gottipati [:aravind]]

The push log isn't working.  Here is the stack trace.

 -->  --> -->

Traceback (most recent call last):
  File "/repo/hg/webroot/mozilla-central/index.cgi", line 28, in ?
    wsgicgi.launch(wsgiapplication(make_web_app))
  File "/usr/lib/python2.4/site-packages/mercurial/hgweb/wsgicgi.py", line 65, in launch
    application(environ, start_response)
  File "/usr/lib/python2.4/site-packages/mercurial/hgweb/request.py", line 100, in run_wsgi
    return application(env, respond)
  File "/usr/lib/python2.4/site-packages/mercurial/hgweb/hgweb_mod.py", line 149, in __call__
    self.run_wsgi(req)
  File "/usr/lib/python2.4/site-packages/mercurial/hgweb/hgweb_mod.py", line 221, in run_wsgi
    method(self, req)
  File "/repo/hg/extensions/pushlog-feed.py", line 46, in pushlogFeed
    (e, urlbase, reponame) = pushlogSetup(web, req)
  File "/repo/hg/extensions/pushlog-feed.py", line 31, in pushlogSetup
    e = lastNEntries(pushdb, 10)
  File "/repo/hg/extensions/pushlog-feed.py", line 21, in lastNEntries
    conn = sqlite.connect(pushdb)
  File "/usr/lib/python2.4/site-packages/mercurial/demandimport.py", line 74, in __getattribute__
    self._load()
  File "/usr/lib/python2.4/site-packages/mercurial/demandimport.py", line 46, in _load
    mod = _origimport(head, globals, locals)
ImportError: No module named sqlite3


I reverted the pushlog to the old one.

Comment 11

[Aravind Gottipati [:aravind]]

Note, I do have python-sqlite2 installed on the boxes.  The pushlog db convert script worked.  The problem seems to appear when running this stuff through the hgweb cgi stuff in python.

Comment 12

[(not currently active) Ted Mielczarek]

Did you get this sorted out? The pushlog appears to be working for me. I updated the hgpoller code at the same time as the other code, so you should be able to pull the latest code from that repo.

Comment 13

[Aravind Gottipati [:aravind]]

Nope, this is still a problem.  I rolled this stuff back to what it was earlier (per comment 10).  I did have the new hgpoller (with the try: import sqlite3, stuff in it).  It appears the mercurial demandimport thingy doesn't work the same as a regular python script.

Comment 14

[(not currently active) Ted Mielczarek]

Ok, I've pushed a change to the hgpoller repo that should fix the demandimport problem. I've also updated the conversion script in the hghooks repo so that it's safe to run again, it will just ignore entries in the flat file that already exist in the db. You should be able to update pushlog-feed.py, verify that it works, and then re-run the conversion script for each repo to pull the data from the flat files that has been added in the interim.

Comment 15

[Aravind Gottipati [:aravind]]

Done.  It seems to all be working now.  Thanks for being patient with this.