[Moztrap] Install python lxml on moztrap nodes

RESOLVED INCOMPLETE

Status

RESOLVED INCOMPLETE
3 years ago
3 years ago

People

(Reporter: peterbe, Unassigned)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/857] )

(Reporter)

Description

3 years ago
We've recently added a new compiled dependency to Moztrap. https://github.com/mozilla/moztrap/commit/10ae3a8beac7699672cdc06502d4a0df4eb0895c#diff-b73d5ed438803e966d21a103301d2376R11

The `requirements/compiled.txt` doesn't get installed, as far as I know, as part of the deployment and any binaries dependencies is something you people have to install for us. 

Can you install lxml 3.4.2 https://pypi.python.org/pypi/lxml/3.4.2 please?
(Reporter)

Comment 1

3 years ago
PR the production server already has a dependency on this but notices if it's available during import time. 

If lxml is not available you can't use XML in the REST API. 

So, after you've installed this can you please kick dev and stage and we'll test that the REST API with XML works.

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/857]

Comment 2

3 years ago
So... this is slightly tricky.

moztrap is not currently deployed in a virtualenv, so there's no good place to install this right now.

At the system level, python-lxml-2.2.3 is already installed. If that version will work, you should be good to go already. If that won't work, then we have basically 3 options that I can see, and which we do largely depends on your own timelines:


1) Punt this. As I understand it, moztrap is due for a refresh/redesign because it's too slow... is this part of that work? If not, maybe we just don't bother. Out of curiosity, why XML, anyway? I though people preferred JSON these days...

2) Set this up in a virtualenv. Probably not too much work if we limit the virtualenv to just the compiled dependencies for now, and let vendor keep handling the non-compiled ones. I would suggest we got the bedrock route and use "peep" with hashes in the requirements/compiled.txt file though... OpSec prefers that since PyPi doesn't enforce any sort of package signing. This will take some work for both of us, most likely- paths in Chief scripts and cron jobs will need to change, for example.

3) Wait for AWS. Everything there will be isolated and deployed entirely differently, so any work spend now is temporary. We can put moztrap near the start of things to migrate. Migrations should start in April, if things go according to plan.
Flags: needinfo?(peterbe)
(Reporter)

Comment 3

3 years ago
Excellent! Maybe.

I don't know how different lxml-2.2 is from lxml-3.4 but it's worth a shot. If it's already installed (on dev, stage AND prod?) we can just do a chief release and see if it works.

1) This is unrelated. I just upgraded django recently which forced me to upgrade tastypie which forces you to install lxml if you want to produce XML. 
Someone in the Taiwan office depends on XML in their REST API output. Why? I have no idea and I think they should consider switching. 

2) Peep or not, we should switch to a virtualenv. Just like we did for airmozilla a couple of weeks ago. Moztrap depends heavily on vendor directories that are checked in to git but I'd be delighted to throw that out in favor of peep but first we need chief to deploy in a virtualenv. 

3) That might be months and months. I think we should first switch to a virtualenv in chief and the rest should follow smoothly. 


For now, I'll get camd to redeploy dev and stage and the Taiwan people can test if lxml 2 is good enough for this version of tastypie.
Flags: needinfo?(peterbe)

Comment 4

3 years ago
Yep, this is already good to go on dev, stage, prod, and the admin node.

NEEDINFO'ing you for the future, for the results of their testing. :)
Flags: needinfo?(peterbe)
Timeout, re-open when this is ready to go.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INCOMPLETE
(Reporter)

Updated

3 years ago
Flags: needinfo?(peterbe)
You need to log in before you can comment on or make changes to this bug.