If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Lots of "Permission denied" errors on /mnt/netapp/air.mozilla.org/*

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: IT-Managed Tools
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: peterbe, Assigned: solarce)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
Created attachment 823343 [details]
traceback.txt

For the last couple of days I've seen a lot of errors like this [1].
That's when it generates a thumbnail from an uploaded image that it stores on the file system as a form of caching. 

What I noticed, over the weekend, was that certain URLs that previously failed now work. Which goes to say that the permission denied errors are sporadic. 

Is there any explanation why we would have permission denied on those file system mounts *sometimes*? And is there any thing we can hack around that?

[1] See attachment.
(Assignee)

Comment 1

4 years ago
(In reply to Peter Bengtsson [:peterbe] from comment #0)
> Created attachment 823343 [details]
> traceback.txt
> 
> For the last couple of days I've seen a lot of errors like this [1].
> That's when it generates a thumbnail from an uploaded image that it stores
> on the file system as a form of caching. 
> 
> What I noticed, over the weekend, was that certain URLs that previously
> failed now work. Which goes to say that the permission denied errors are
> sporadic. 
> 
> Is there any explanation why we would have permission denied on those file
> system mounts *sometimes*? And is there any thing we can hack around that?
> 
> [1] See attachment.

For some reason that folder had the wrong permissions and apache couldn't write to it

[root@genericadm.private.phx1 5a]# ls -lah
total 52K
drwxr-xr-x  11 apache apache 4.0K Oct 16 04:27 .
drwxr-xr-x 258 apache apache  12K May  9 07:59 ..
drwxr-xr-x   2 apache apache 4.0K Jun  4 02:11 0b
drwxr-xr-x   2 apache apache 4.0K Nov 25  2012 1b
drwxr-xr-x   2 apache apache 4.0K Jul 29 14:27 23
drwxr-xr-x   2 apache apache 4.0K Oct 12  2012 31
drwxr-xr-x   2 apache apache 4.0K Jul 16 18:50 4f
drwxr-xr-x   2 apache apache 4.0K Oct 16 04:27 5b
drwxr-xr-x   2 root   root   4.0K May 20 15:06 9a
drwxr-xr-x   2 apache apache 4.0K Jul 10 10:56 a9
drwxr-xr-x   2 apache apache 4.0K Aug 21 11:05 cc
[root@genericadm.private.phx1 5a]# ls -lah 9a/
total 84K
drwxr-xr-x  2 root   root   4.0K May 20 15:06 .
drwxr-xr-x 11 apache apache 4.0K Oct 16 04:27 ..
-rwxr-xr-x  1 root   root    69K May 20 15:06 5a9aab1e6f84bb2c8580635841bb65ff.png
[root@genericadm.private.phx1 5a]# chown -R apache:apache 9a
[root@genericadm.private.phx1 5a]# ls -lah 9a/
total 84K
drwxr-xr-x  2 apache apache 4.0K May 20 15:06 .
drwxr-xr-x 11 apache apache 4.0K Oct 16 04:27 ..
-rwxr-xr-x  1 apache apache  69K May 20 15:06 5a9aab1e6f84bb2c8580635841bb65ff.png

I've fixed them

Nothing else seems to be owned by root, so you should be good

[root@genericadm.private.phx1 media]# find . -user root
./push_log
./revision
Assignee: server-ops-webops → bburton
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

4 years ago
That would explain the sporadic nature. The hashes of the thumbnails get "sharded" up and for it starting on 9a we were unlucky. 

I think we can resolve this bug now. I'm guessing how it got like that was some accidentally run apache back in May that caused it to run as root and create ./9a back then.
You need to log in before you can comment on or make changes to this bug.