375800 - AMO upload problems

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Scott [:fligtar]]

I've seen several reports today of errors uploading extensions/previews to AMO. In both extension and preview uploads, the error returned is an AMO error that occurs when the uploaded file name is empty (ie, $_FILES['uploadedfile']['name']). The HTTP upload error ($_FILES['uploadedfile']['error']) is checked after that point, so it doesn't get far enough for me to find out any more information.

The test suite shows that all that paths that the app needs to be writable are writable (except for the php temp dir, I guess).

We could use some help debugging this - I first heard reports of this this afternoon, possibly related to bug 375706? I think that most uploads are working, but it seems that there are some that aren't.

Comment 3

[Justin Scott [:fligtar]]

I've seen about 15 reports of this now on IRC, bugs, newsgroups, and conversations. Several people have tried the same file again later and had it work, and it works fine in testing (no code changes have been made in that area) so I'm fairly sure it's a production problem.

Comment 4

[Aaron]

Tried upload on Firefox, IE and Konqueror and on Windows and Linux, so the problem is definitely not client-side. It just says "Please upload a file." every time I try.

Comment 5

[Aaron]

By "client-side" I meant to say that it is not (as someone who's upload worked on Safari claimed on Mozillazine) a Firefox specific bug.

Comment 6

[Aaron]

My upload just worked

Comment 7

[Aravind Gottipati [:aravind]]

It appears that these upload issues are not consistent.  clouserw thinks they may be related to the blank page problems (apache segfaults) we have been noticing.  He has also confirmed that they have been able to upload content most of the times.

I would like to get the apache servers to quit segfaulting first and then take a look at this issue.

Also, if you are still having this problem, please provide the exact time it happens.  This will help us correlate this to backend server problems (if they are indeed related).

Comment 8

[Nils Maier [:nmaier]]

Attachment: some Icon change/add attempts

Comment 9

[Nils Maier [:nmaier]]

I actually fail to upload stuff most of the times.
Maybe 1 in 4 attempts succeed (only tried icon).
Fellow dTa developer has the same problems.

Just some random thoughts:
* May it be related to caching/memcached?
This would at least make some sense to me.
* You do not actually check _FILES but that Cake data/model array. Maybe it is a bad mapping (due to caching?). Ever tried comparing the values from _FILES and cake? Ever actually watched the codepath it takes to contruct the cake provided values.
Filename isn't actually "empty".
* Segfaults? Didn't see any. Everything worked (that success/redirection message, and the 'old' icon was served.)
* Might be a memory related issue (hitting either memory_limit or some system limit). But then again this wouldn't explain why stuff goes successful. Such a failure would be fatal, so execution would stop there.

Sorry for bugspam, and good luck. Hope my log helps.
If you need further information mail me under the address of this account.

Comment 10

[Aravind Gottipati [:aravind]]

okay, based on what I see in the timestamps in that attachments, this is
probably not related to the apache segfaults.

one possibility was that the uploads may be failing because consecutive
requests do not end up on the same server, but clouserw and flitgair assured me
that this wasn't necessary.

Continuing to look..

Comment 11

[DonGato]

I'm having this issue now and since a couple of hours ago (tried 6 times already an no success). I'm trying to update Whitehart TB.

Comment 12

[Aravind Gottipati [:aravind]]

try uploading to preview.addons.mozilla.org.  preview points to only one server (so it makes it easier to troubleshoot).  Note that it has its own db instance, so your changes will not be visible in the production addons.

Comment 13

[Aravind Gottipati [:aravind]]

Actually, Jeremy may have fixed the problem.  He found incorrect perms on /tmp (on all the servers) - which was caused by me :(

He fixed the problems and other php applications having upload issues are now working fine.  Please retest your uploads with the regular addons site and let us know if the uploads go through okay.

Comment 14

[DonGato]

It seems it's working now.