564185 - [amo] File permission problem with pip updates

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jeff Balogh (:jbalogh)]

I've been seeing this since 12:00pm yesterday:

Cron <apache@mradm02> rm -rf /data/virtualenvs/zamboni/build && /data/virtualenvs/zamboni/bin/pip -q install -U -i https://hudson.mozilla.org/pypi/ -r https://hudson.mozilla.org/pypi/pip-manifest.txt > /dev/null

Traceback (most recent call last):
 File "/data/virtualenvs/zamboni/bin/pip", line 3, in <module>
   sys.exit(pip.main())
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/__init__.py", line 75, in main
   return command.main(initial_args, args[1:], options)
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/basecommand.py", line 130, in main
   log_fp = open_logfile_append(log_fn)
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/basecommand.py", line 182, in open_logfile_append
   log_fp = open(filename, 'a')
IOError: [Errno 13] Permission denied: './pip-log.txt'

Comment 1

[Wil Clouser [:clouserw]]

I'll bet the dir is owned by root.  It should be owned by apache.

Comment 2

[Shyam Mani [:fox2mike]]

I'll bet if you cleaned up the cruft left by pip in /tmp, you won't run into these issues again! :)

Comment 3

[Shyam Mani [:fox2mike]]

17G     total <- /tmp

Wiping it now.

Comment 4

[Jeremy Orem [:oremj]]

Looks clean now. I thought the new version of pip (0.7.1) was supposed to clean up after itself.

Comment 5

[Jeremy Orem [:oremj]]

I've also chowned the virtualenv to apache, so this should work again.

Comment 6

[Ian Bicking (:ianbicking)]

Could this just be leftover cruft from an older pip?  0.7.1 includes tests for temporary file cleanup, so I feel somewhat confident it really is cleaning things up now.

Also as of pip 0.7 it should be putting the log file in ~/.pip/pip.log so that error seems indicative of an older pip being used.

Comment 7

[Jeremy Orem [:oremj]]

Yeah, I misunderstood which box this was happening on.

Comment 8

[Jeff Balogh (:jbalogh)]

Traceback (most recent call last):
 File "/data/virtualenvs/zamboni/bin/pip", line 3, in <module>
   sys.exit(pip.main())
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/__init__.py", line 94, in main
   return command.main(initial_args, args[1:], options)
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/basecommand.py", line 139, in main
   log_fp = open_logfile(log_fn, 'w')
 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/pip/basecommand.py", line 194, in open_logfile
   os.makedirs(dirname)
 File "/data/virtualenvs/zamboni/lib64/python2.6/os.py", line 157, in makedirs
   mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/www/.pip'

Comment 9

[Jeff Balogh (:jbalogh)]

That traceback was from 10:00am.

Comment 10

[Ian Bicking (:ianbicking)]

$HOME should be set to a writable location (/tmp is fine)

Comment 11

[Ian Bicking (:ianbicking)]

I opened a ticket in pip too (though I'd recommend just setting $HOME now): http://bitbucket.org/ianb/pip/issue/116/opening-log-file-shouldnt-fail

Comment 12

[Jeff Balogh (:jbalogh)]

This is happening once an hour.  I don't think it's preview or next.  Is it the box that runs cron jobs?

Comment 13

[Jeremy Orem [:oremj]]

That's coming from:
0 * * * * rm -rf /data/virtualenvs/zamboni/build && /data/virtualenvs/zamboni/bin/pip -q install -i https://hudson.mozilla.org/pypi/ -r https://hudson.mozilla.org/pypi/pip-manifest.txt > /dev/null


You'll just need to export HOME=/tmp in that cronjob.

Comment 14

[matthew zeier [:mrz]]

Not sure if there's any IT action here, closing.

Comment 15

[Wil Clouser [:clouserw]]

Done in http://github.com/jbalogh/zamboni/commit/88b6ee3fed6196cf5cd2796d2bca87ce7ba08c6d

The IT action here is to make sure it's using /tmp and not filling up the disk, I guess.

Comment 16

[Jeremy Orem [:oremj]]

Looks fine so far.