1071019 - Symbol upload fails for telenor: internal server error

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Benjamin Smedberg]

Moved from bug 1061482, Andreas Pehrson from Telenor says:

===
Benjamin, FYI I just got a 500 Internal Server Error when trying to upload symbols on https://crash-stats.mozilla.com/symbols/upload/web/ using the account in comment 13. Could you take a look?

The following headers in the response look like they could be useful:
Server:"Apache"
X-Backend-Server:"socorro4.webapp.phx1.mozilla.com"
X-Frame-Options:"DENY"
===

The account for upload is [e-mail].

peterbe, are you the right person to investigate/fix?

Comment 1

[Peter Bengtsson [:peterbe]]

It failed because of permission problems on the server. 
https://errormill.mozilla.org/webtools/socorro-prod/group/172759/

I'm going to email phrawzty too to make sure he understands the urgency of it.

Comment 2

[Daniel Maher [:phrawzty]]

Looks like the django symlink is busted for some reason.  Investigating.

ಠ_ಠ

Comment 3

[Daniel Maher [:phrawzty]]

On all the Prod webheads:
* Moved content from webapp-django/media/symbols_upload to the mount.
* Removed said errant directory.
* Created symlink from webapp-django/media/symbols_upload to the mount.

The error condition should be resolved; however, it's troubling that the symlink wasn't there in the first place.  This will require further investigation.

Comment 4

[Andreas Pehrson [:pehrsons]]

I just made another attempt at uploading the symbols. Still getting 500 internal server error.

Daniel, could you have a quick look to see if it's the same issue or something new?

Comment 5

[Daniel Maher [:phrawzty]]

Indeed it does appear to be exactly the same problem.  Prod was pushed yesterday after I applied the manual fix in comment #3, which provides further evidence that the pushes themselves are the source of the problem.  In any case, I will apply the manual fix again - please see bug 1071062 for further analysis.

Comment 6

[Lonnen :lonnen]

Prod was not pushed yesterday, to my knowledge.

Comment 7

[Daniel Maher [:phrawzty]]

From my IRC logs:
> Day changed to 22 Sep 2014
> 11:05:43 <@lonnen> objections to shipping some code?
> 11:06:58 < peterbe> none from me

??

Comment 8

[Lonnen :lonnen]

I got pulled away to work on other things and it didn't happen. There was no push bug, email about new code, etc.

Comment 9

[Daniel Maher [:phrawzty]]

I see - well that may change things somewhat, and indeed, the symlink has been replaced by a directory. Again.

Comment 10

[Andreas Pehrson [:pehrsons]]

I also tried again earlier to no avail (another 500), hoping you had fixed it manually phrawzty. It was 5-6 hours ago.

Any chance my uploads actually trigger this bug?

Comment 11

[Daniel Maher [:phrawzty]]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #10)
> I also tried again earlier to no avail (another 500), hoping you had fixed
> it manually phrawzty. It was 5-6 hours ago.
> 
> Any chance my uploads actually trigger this bug?

Yes, the problem is recurring, and I'm literally investigating it right now.  I don't know if your uploads are triggering the condition or not, though that's an interesting idea that I will explore further.

Comment 12

[Lonnen :lonnen]

Had to put the investigation on hold for the night. Phrawzty narrowed it down to a cron job that is clobbering the symlinks every 5 minutes or so.

Will come back to it tomorrow.

Comment 13

[Daniel Maher [:phrawzty]]

The root cause of this issue has been identified and addressed - see bug 1071062.  Closing.

Comment 14

[Andreas Pehrson [:pehrsons]]

Managed to upload now. Thanks Daniel!