442055 - Replace user-storage flat files in Pootle with database

Status: RESOLVED FIXED

(Webtools Graveyard :: Verbatim, defect, P2)

Description

[Dan Schafer [:dschafer]]

Currently, Pootle stores a lot of information in flat files (overall preferences, language info, project info, user preferences).  These should be stored in databases instead.

Comment 1

[Dan Schafer [:dschafer]]

Note that currently, the .prefs file can't use dots in usernames; hence, the dots in LDAP lookups are replaced with D0T.  This is a terrible, terrible, terrible hack, and needs to be fixed in the database conversion.  grep-ing for "D0T" in the pootle directory should find all instances of this hack.

Comment 2

[Dan Schafer [:dschafer]]

Attachment: Initial version of patch moving user info to database

This patch will relocate the user storage information to a database rather than a flat file.  It is mostly working right now, though occasionally this error occurs:
  TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30

More testing also needs to be done to make sure all features even remotely reliant on the user storage are still working, but the major features (login, registration, admin account creation) all appear to work.

Comment 3

[Dan Schafer [:dschafer]]

Attachment: Improved DB users storage

This patch improves the database storage of users.  The QueuePool bug has been fixed from the previous version, and a few more obscure features relying on the flat files have been updated.  Some rather important performance issues aside (Suggest and Submit take almost 10 seconds, for some reason), the functionality of users databases is pretty much done.

Comment 4

[Dan Schafer [:dschafer]]

The performance issues are now addressed in bug 446339, since they appear to be independent of this bug.  Since that issue has nothing to do with this patch, I'll do some thorough testing of this patch tomorrow, then hopefully commit.

Comment 5

[Dan Schafer [:dschafer]]

Attachment: Uses SQLAlchemy to move users storage to database

This is in revision 7816.

Comment 6

[Dan Schafer [:dschafer]]

Everything seems to be working with users in a database, so marking this FIXED.

The previous attachment modified a function (getuserswithrights in projects.py) to deal with dots in usernames and the existence of .prefs files to store rights.  Bug 446441 was created to deal with moving rights/goals to a database, and a note was made there about this issue.

I changed this bug to deal specifically with moving users to a database (since I accidentally had been using it for that), and created bug 446441 for rights/goals and 446442 for projects/languages.  Bug 446443 was created to track all flat file -> database conversions.