852678 - Can't deploy to paas.allizom.org a flask app that has bztools requirements: error: could not create '/opt/ActivePython-2.7/lib/python2.7/site-packages/scripts': Read-only file system

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Output from running the post-staging hook to install bztools

See attachment for the output - basically I have a relatively simple Flask app (https://github.com/mozilla/relmandash) that requires a package to help with bugzilla API calls (https://github.com/mozilla/bztools) and I've tried a lot of combinations in pip however the bztools package has to be installed separately from the other dependencies or it cannot find its dependent modules. So in my stackato.yml I put a post-staging hook to install bztools separately and got the output attached that says I cannot install due to some read-only stuff.

Getting this up and running is blocking an intern project that we are trying to finish before the end of the month - so we're quite eager to find a way to get past this.

Comment 1

[Jake Maul [:jakem]]

I just tried to do this myself, simply cloning from your repo, tweaking the names in stackato.yml (so as to use my own instance), and pushing.

If I'm reading this error right, the problem is that the bztools-dev package can't find one of its requirements, remoteobjects. However, as far as I can tell that's perfectly correct... remoteobjects hasn't been installed. remoteobjects-dev was. Different package name.

staging:        Downloading/unpacking remoteobjects-dev from git+git://github.com/mozilla/remoteobjects.git@beba672b9b1f9fc53d526242ce92d1e050a343b9 (from -r /staging/staged/app/requirements.txt (line 2))
staging:          Cloning git://github.com/mozilla/remoteobjects.git (to beba672b9b1f9fc53d526242ce92d1e050a343b9) to ./build/remoteobjects-dev
staging:          Could not find a tag or branch 'beba672b9b1f9fc53d526242ce92d1e050a343b9', assuming commit.
staging:          Running setup.py egg_info for package remoteobjects-dev
...
...
staging:   Running setup.py egg_info for package bztools-dev
staging:     Traceback (most recent call last):
staging:       File "<string>", line 14, in <module>
staging:       File "/staging/staged/build/bztools-dev/setup.py", line 12, in <module>
staging:         version=__import__('bugzilla').__version__,
staging:       File "bugzilla/__init__.py", line 4, in <module>
staging:         from remoteobjects import http
staging:     ImportError: No module named remoteobjects
staging:     Complete output from command python setup.py egg_info:
staging:   File "<string>", line 14, in <module>
staging:   File "/staging/staged/build/bztools-dev/setup.py", line 12, in <module>
staging:     version=__import__('bugzilla').__version__,
staging:   File "bugzilla/__init__.py", line 4, in <module>
staging:     from remoteobjects import http
staging: ImportError: No module named remoteobjects

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

I haven't pushed the configuration I am using right now - still troubleshooting - but here's what I'm doing - I took bztools out of requirements.txt and added:

hooks:
  post-staging:
    - pip install -r more-requirements.txt

to the stackato.yml

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

and the reason I'm doing the workaround in comment 2 is because of the error you are getting in comment 1 - which seems to be due to pip not recognizing that it has in fact installed a dependency higher up in the requirements doc *during* installation.  Hence the need to do bztools separately.

Comment 4

[Jake Maul [:jakem]]

It definitely should recognize a dependency higher up in the installation order. I'm 99% sure that what's going on here is that it's installing remoteobjects-dev... bztools-dev trying to use remoteobjects. Not the same package that it installed. That will never work, regardless of how pip operates. :)

The post-staging hook is probably not going to be feasible. We could open this bug upstream, but there's virtually no chance they'll take action on it. They'll most likely get back to us in a couple days and say "use requirements.txt or a requirements block in stackato.yml for that". Until the naming issue is sorted out, we can't really show that this doesn't work.

Comment 5

[Kyle Lahnakoski [:ekyle]]

Lukas, I had the same problem on my machine.  I eventually found the correct combination of installs.  
    1) install the original bztools (I put it in bztools2), then 
2) install Lianne's bztools, and finally install relmandash

Comment 6

[Kyle Lahnakoski [:ekyle]]

Lukas, I had the same problem on my machine.  I eventually found the correct combination of installs.  
    1) install the original bztools (I put it in bztools2), then 
    2) install Lianne's bztools, and finally
    3) install relmandash


## MAKE NEW PYTHON ENVIRONMENT
cd C:\Users\klahnakoski\git\relmandash
CALL virtualenv relmandashenv

## ACTIVATE ENVIRONMENT
cd C:\Users\klahnakoski\git\relmandash
CALL relmandashenv\Scripts\activate       ##USE deactivate TO LEAVE VIRTUAL ENV

## MUST INSTALL BZTOOLS FROM git://github.com/mozilla/bztools.git FIRST
cd C:\Users\klahnakoski\git\bztools2
CALL pip install -r requirements.txt

## MUST INSTALL BZTOOLS FROM git://github.com/lianne719/bztools.git SECOND
cd C:\Users\klahnakoski\git\bztools
CALL pip install -r requirements.txt

## remoteobjects IS NOW INSTALLED, PROCEED WITH relmandash INSTALL
cd C:\Users\klahnakoski\git\relmandash
CALL pip install -r requirements.txt

## RUN !! :)
CALL python runserver.py

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Kyle Lahnakoski from comment #6)
> Lukas, I had the same problem on my machine.  I eventually found the correct
> combination of installs.  
>     1) install the original bztools (I put it in bztools2), then 
>     2) install Lianne's bztools, and finally
>     3) install relmandash
> 
> 
> ## MAKE NEW PYTHON ENVIRONMENT
> cd C:\Users\klahnakoski\git\relmandash
> CALL virtualenv relmandashenv
> 
> ## ACTIVATE ENVIRONMENT
> cd C:\Users\klahnakoski\git\relmandash
> CALL relmandashenv\Scripts\activate       ##USE deactivate TO LEAVE VIRTUAL
> ENV
> 
> ## MUST INSTALL BZTOOLS FROM git://github.com/mozilla/bztools.git FIRST
> cd C:\Users\klahnakoski\git\bztools2
> CALL pip install -r requirements.txt
> 
> ## MUST INSTALL BZTOOLS FROM git://github.com/lianne719/bztools.git SECOND
> cd C:\Users\klahnakoski\git\bztools
> CALL pip install -r requirements.txt
> 
> ## remoteobjects IS NOW INSTALLED, PROCEED WITH relmandash INSTALL
> cd C:\Users\klahnakoski\git\relmandash
> CALL pip install -r requirements.txt
> 
> ## RUN !! :)
> CALL python runserver.py

This is for local running, sure, and I haven't had a problem deploying it locally in a virtualenv with bztools installed.  What I'm trying to do is push it to paas.allizom.org and that is where the problem currently lies.

Comment 8

[Jake Maul [:jakem]]

A lot has happened via IRC. Here's a summary.

My previous comment re: pip installation ordering / dependency tracking appears to be incorrect. When installing packages via requirements.txt, pip seems to consider each line in isolation- it does not attempt to satisfy a dependency with a another file in the requirements.txt. This is usually not a problem, because it *does* follow dependencies recursively. So, if you need bztools, and bztools needs remoteobjects, and remoteobjects needs httplib2, all of it will be installed.

However, as it happens bztools actually seems to be slightly broken in this manner. When you install bztools via pip, with nothing else yet installed, it does not try to pull in remoteobjects. This happens even outside of Stackato... it's not a PaaS problem, it's a package dependency tracking / installation one. When installing bztools, it will not automatically pull in remoteobjects as a dependency.

remoteobjects doesn't have this issue... it pip installs cleanly in an empty environment.


Lukas was going to look into making bztools' installation to make this go away.


Side note: we also saw a lot of weirdness with respect to which package was reported as failing. I'm not sure yet what causes this, but I think this is pip may be behaving oddly in the case where a package messes up. One way to test would be to omit bztools and see if it pushes cleanly. When I do this, it stages without issue. It fails to actually start because it can't import from BMOAgent (presumably provided by bztools), but at least all the other problems with staging seem to go away. :)

Comment 9

[Ashish Vijayaram [:ashish]]

(dropping severity to avoid paging oncalls since this is actively being worked upon)

Comment 10

[Jake Maul [:jakem]]

I don't think there's anything for us (webops) to do here. Investigation in comment 8 indicates that this is a problem with bztools' upstream dependency listing, rather than one with PaaS.

I'm going to close this out, but if there is additional work I have overlooked, please reopen with details. Thanks!