Closed Bug 523920 Opened 15 years ago Closed 15 years ago

Move Verbatim to production server from sm-cms01

Categories

(mozilla.org Graveyard :: Server Operations, task)

All
Other
task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: sethb, Assigned: zbraniecki)

References

()

Details

I want to move the Verbatim work that Gandalf and Wenzel are doing into production.  

*  Presently, the test environment lives on https://sm-cms01.mozilla.org:8081/ 
*  We want LDAP authentication for our localization team members to be able to log in.  All of our locale leaders have LDAP credentials.  
*  I believe we will need SSL, which might involve certificates.  
*  We can start with a RHEL VM
*  The machine would provide the entire localization community access to the translation tool.  Right now we have 75 locales, but all on different timezones and not all of them accessing at once.  But, the high end estimate is that as many as 100 people could be accessing the machine at one time.
*  Ideally, we would have a nice name and it would love on *.mozilla.org.  The leading suggestion is localize.mozilla.org  

(NB:  at some point in the future, we hope to expand localize.mozilla.org into a much larger Mozilla l10n portal)
Assignee: server-ops → thardcastle
Pootle is not the easiest software to deploy, so we'd like to work with IT on getting it right without making it too much guesswork for them.

The best way to do it is probably: You set us up with a VM/box that we can eventually point a public domain name to, and I install a Pootle instance on it. That'd allow us to apply patches to Pootle when they are necessary and fix problems with upcoming translation projects, or when somethings broken with the data Pootle works on. Since this would be in production, we'd still ask IT to keep this server in the loop of their regular security updates for all non-Pootle packages on that box, but webdev can take care of the application itself.

If IT does not like that, I can offer making a python virtualenv containing both the requirements as well as the tool itself, if I can manage to make one that's portable to a production server/VM. We'd need an Apache instance with mod_python, a "verbatim" user account that the software runs as, etc., and due to Pootle's occasional stubbornness, this would likely cause the occasional headache until it's all up and running properly.

Please let us know what you prefer. Also, I marked this bug dependent on bug 524499, because I'll need to be able to SSH into sm-cms01 in order to look up our staging instance's configuration.
Actually, after discussing this in the Verbatim meeting, the best thing might indeed providing a virtualenv, so IT can properly deploy it, while allowing to `svn up` the application source when necessary.
It's important that we have decent access to logs, etc. for debugging.
Out of curiosity, any progress on this?  I'd really like to start using Verbatim/Pootle for a few projects.  It's a quarterly goal for l10n to get this into production, so we are eager to move this forward so we can experiment with other things we'd like to do with Verbatim.  

How may I help?  Do you need anything from me or further clarification on the needs we have?
I have been (and still am) working on making a package (or at least writing deployment instructions) for Pootle. The latest version of Pootle contained a bug in committing to SVN, which they needed to fix first.

Once we got that skeleton on a box somewhere, it'd be helpful if Gandalf could assist in configuring that Pootle instance with regard to caching etc.
All right, I updated the deployment instructions here:
https://wiki.mozilla.org/Verbatim:Deployment

All config files I am referring to on the wiki page are in [https://gist.github.com/cb33a48c6945db24f13a this "gist"] on github.

Please try setting up an instance like that, and get back to us if there are open questions or errors anywhere that I did not see when trying all this on khan.

By the way, re: SVN instructions, you'll find the SSH key on sm-cms01 (verbatim user) and also the hooks have already been deployed to our SVN server and should still be in place there.

Note: We *will* need project checkouts (like the amo one mentioned as an example in the deployment instructions), but let's get the plain instance up and running first, then get projects in place.
Severity: normal → major
Trevor: Did you make a happy little VM for this yet?
Can we get an ETA/Update on this?
(In reply to comment #8)
> Can we get an ETA/Update on this?

Just got the VM built. Worked through most of the Verbatim:Deployment instructions but ran into some problems. ETA: Tomorrow.
This is nearly there. LDAP/SVN integration isn't working yet, might be able to get that finished tonight.

Does this mean something is broken?
./manage.py refresh_stats
...for each locale...
2009-11-13 03:57:47,801 WARNING Could not initialize indexer for terminology in zh_CN: Indexer: no indexing engines are available
2009-11-13 03:57:47,805 INFO Updating stats for terminology/zh_CN/gnome/zh_CN.po
2009-11-13 03:57:47,809 DEBUG cache miss for /data/www/localize.mozilla.org/verbatim/pootle_env/Pootle-1.3.0-beta4/po/terminology/zh_CN/gnome/zh_CN.po
Thanks for working on this, Trevor. No, these warnings do not mean something is broken. Ideally, we'd use an indexer engine such as Apache Lucene, but they are hard to set up, so we'll go without them for now.
(In reply to comment #10)
> This is nearly there. LDAP/SVN integration isn't working yet, might be able to
> get that finished tonight.

Hey Travor:  any update on this?  Thanks!
did you look at the verbatim instance in /home/verbatim on sm-cms01? At least the LDAP settings should be there.
L10n would like to start using this for localizing getpersonas next Monday. We should work together to iron out remaining problems so they can start using it.
I just got the LDAP config worked out from sm-cms01 and had Aravind look it over. SVN and SSL setup are the last steps. Working on those now.
That's good. Thanks for the update!
chizu, can you update with some status/progress?  This is a Q4 objective for l10n and I'm sure they'd like to know where we are with this and a sense of ETA.
Site is up at http://localize.mozilla.org/ but SSL isn't working (netscaler config issue) and the SVN setup doesn't exist yet. ETA: Today.
https://localize.mozilla.org/ is up and SVN commits should work. Please verify.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
(In reply to comment #19)
> https://localize.mozilla.org/ is up and SVN commits should work. Please verify.

Site it up, but why don't we see the "Verbatim" instance of Pootle.  This looks like a Pootle install.  

I don't see AMO or SUMO under the project list.  

This looks like a straight Pootle install and not the instance that Fred/Wil/gandalf have been working on, right?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
LDAP works, but again, I log into a Pootle server instnace and not what we have been calling "Verbatim".
Yes, the next step is actually getting projects into this Pootle instance. Probably AMO and getpersonas first, then SUMO? Although the SUMO team plans on switching to .po files, which is why we might want to hold off on this until they do.

It might make sense to file new bugs on these things though.
(In reply to comment #22)
> Although the SUMO team plans on
> switching to .po files, which is why we might want to hold off on this until
> they do.

We plan on making the switch, but we have no idea when. I was hoping for Q1 but the plans keep shifting.
(In reply to comment #22)
> Yes, the next step is actually getting projects into this Pootle instance.
> Probably AMO and getpersonas first, then SUMO? 

OK.  Any chance to get this set up for Monday morning PST?

> Although the SUMO team plans on
> switching to .po files, which is why we might want to hold off on this until
> they do.

I keep hearing about this.  I've been pressed by dtenser to launch Verbatim for SUMO.  But, if we are changing the site to gettext, then I guess we wait?  Insight/recommendation from the SUMO team would be cool.
(In reply to comment #24)
> I keep hearing about this.  I've been pressed by dtenser to launch Verbatim for
> SUMO.  But, if we are changing the site to gettext, then I guess we wait? 
> Insight/recommendation from the SUMO team would be cool.

See comment 23.
(In reply to comment #25)
> (In reply to comment #24)
> > I keep hearing about this.  I've been pressed by dtenser to launch Verbatim for
> > SUMO.  But, if we are changing the site to gettext, then I guess we wait? 
> > Insight/recommendation from the SUMO team would be cool.
> 
> See comment 23.

Yeah, I read comment 23.  I guess I was trying to elicit a bit more certainty from the SUMO project managers since we have been pushed to get Verbatim live.  The "but we have no idea when" part of comment #23 was what I was hoping to hear more about from dtenser and others.   :)  

Perhaps it is another set of bugs, as mentioned in comment #22.
What about the implementation of SUMO support that was done earlier this year for the non-gettext version (language.php)? It seemed to be working fairly well at the time.

James mentioned in one SUMO dev meeting that changing to .po files would be fairly straightforward, but we're currently in a pretty significant effort to upgrade to Tiki 5.1, which should be completed before we start the .po project upstream. 

Upgrading to TW5.1 is a Q1 goal and l10n would be a great next step after it. Depending on how long it takes to complete the upgrade, .po implementation would start somewhere between March and April.
Blocks: 530492
I'm taking this bug over now.

I'll set up AMO and SUMO on the new instance and customize it. I'm going use the hooks that we have for SUMO for now, and when SUMO will switch to gettext, we will follow.
Assignee: thardcastle → gandalf
(In reply to comment #28)
> I'll set up AMO and SUMO on the new instance and customize it. I'm going use
> the hooks that we have for SUMO for now, and when SUMO will switch to gettext,
> we will follow.

Thank you. Do you have shell-level access to the box it's running on though? If so, and if IT is fine with it, you would be able to fix bug 530492 too.
Blocks: 530493
(In reply to comment #27)
> What about the implementation of SUMO support that was done earlier this year
> for the non-gettext version (language.php)? It seemed to be working fairly well
> at the time.

I have some concern on the forthcoming need to convert the existing SUMO translations to .po files.  If we could just start SUMO with .po for our Verbatim launch, it may make life easier for Gandalf and Fred in the future.  But, looks like that is months away, so we'll move forward as planned with the existing SUMO setup.
Blocks: 530769
If the current SUMO implementation mostly works and it's just a matter of adjusting it for the latest Verbatim version, moving forward as planned would be much appreciated. Thanks Gandalf/Fred for working on this!
(In reply to comment #28)
> I'm taking this bug over now.
> 
> I'll set up AMO and SUMO on the new instance and customize it. I'm going use
> the hooks that we have for SUMO for now, and when SUMO will switch to gettext,
> we will follow.

Gandalf:  Can we get Get Personas hooked in, too?
hooked getpersonas!
Status: REOPENED → RESOLVED
Closed: 15 years ago15 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.