931795 - Lots of "Permission denied" errors on /mnt/netapp/air.mozilla.org/*

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Peter Bengtsson [:peterbe]]

Attachment: traceback.txt

For the last couple of days I've seen a lot of errors like this [1].
That's when it generates a thumbnail from an uploaded image that it stores on the file system as a form of caching. 

What I noticed, over the weekend, was that certain URLs that previously failed now work. Which goes to say that the permission denied errors are sporadic. 

Is there any explanation why we would have permission denied on those file system mounts *sometimes*? And is there any thing we can hack around that?

[1] See attachment.

Comment 1

[Brandon Burton [:solarce]]

(In reply to Peter Bengtsson [:peterbe] from comment #0)
> Created attachment 823343 [details]
> traceback.txt
> 
> For the last couple of days I've seen a lot of errors like this [1].
> That's when it generates a thumbnail from an uploaded image that it stores
> on the file system as a form of caching. 
> 
> What I noticed, over the weekend, was that certain URLs that previously
> failed now work. Which goes to say that the permission denied errors are
> sporadic. 
> 
> Is there any explanation why we would have permission denied on those file
> system mounts *sometimes*? And is there any thing we can hack around that?
> 
> [1] See attachment.

For some reason that folder had the wrong permissions and apache couldn't write to it

[[e-mail]1 5a]# ls -lah
total 52K
drwxr-xr-x  11 apache apache 4.0K Oct 16 04:27 .
drwxr-xr-x 258 apache apache  12K May  9 07:59 ..
drwxr-xr-x   2 apache apache 4.0K Jun  4 02:11 0b
drwxr-xr-x   2 apache apache 4.0K Nov 25  2012 1b
drwxr-xr-x   2 apache apache 4.0K Jul 29 14:27 23
drwxr-xr-x   2 apache apache 4.0K Oct 12  2012 31
drwxr-xr-x   2 apache apache 4.0K Jul 16 18:50 4f
drwxr-xr-x   2 apache apache 4.0K Oct 16 04:27 5b
drwxr-xr-x   2 root   root   4.0K May 20 15:06 9a
drwxr-xr-x   2 apache apache 4.0K Jul 10 10:56 a9
drwxr-xr-x   2 apache apache 4.0K Aug 21 11:05 cc
[[e-mail]1 5a]# ls -lah 9a/
total 84K
drwxr-xr-x  2 root   root   4.0K May 20 15:06 .
drwxr-xr-x 11 apache apache 4.0K Oct 16 04:27 ..
-rwxr-xr-x  1 root   root    69K May 20 15:06 5a9aab1e6f84bb2c8580635841bb65ff.png
[[e-mail]1 5a]# chown -R apache:apache 9a
[[e-mail]1 5a]# ls -lah 9a/
total 84K
drwxr-xr-x  2 apache apache 4.0K May 20 15:06 .
drwxr-xr-x 11 apache apache 4.0K Oct 16 04:27 ..
-rwxr-xr-x  1 apache apache  69K May 20 15:06 5a9aab1e6f84bb2c8580635841bb65ff.png

I've fixed them

Nothing else seems to be owned by root, so you should be good

[[e-mail]1 media]# find . -user root
./push_log
./revision

Comment 2

[Peter Bengtsson [:peterbe]]

That would explain the sporadic nature. The hashes of the thumbnails get "sharded" up and for it starting on 9a we were unlucky. 

I think we can resolve this bug now. I'm guessing how it got like that was some accidentally run apache back in May that caused it to run as root and create ./9a back then.