614012 - Website: Service unavailable errors when submitting personas

Status: RESOLVED WONTFIX

(Websites Graveyard :: getpersonas.com, defect, P1)

Description

[Shae Rivard [:crimius]]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; rv:2.0b7) Gecko/20100101 Firefox/4.0b7
Build Identifier: 

An approver just sent me an email abotu how they received a Service Unavailable error when they submitted a persona.  If I'm not mistaken, we're pretty sure it's the mail function timing out that causes these errors.  Most of the time we get them when reviewing personas, but if it's happening when users are submitting personas, it's probably the main reason we get as many duplicate submissions that we do.

Reproducible: Always

Comment 1

[Michael Morgan [:morgamic]]

SMTP info could be switched to use socketlabs instead, if it's just hanging on outbound.

Comment 2

[Michael Morgan [:morgamic]]

Deb - is there an error code described in the email?

Comment 3

[Michael Morgan [:morgamic]]

Assuming 503, we'd be looking at:
http://svn.mozilla.org/projects/getpersonas.com/trunk/server/lib/review.php

I don't know why it's timeout randomly, but here are ideas:
- php.ini on some nodes is not set properly
- hosts file does not have 'localhost' as a valid host, and mail() is using that by default

Possible solutions:
- set up .ini config to use socketlabs explicitly
- add configurations to personas and write wrappers for mail() that don't depend on server configs

Comment 4

[Wil Clouser [:clouserw]]

> An approver just sent me an email abotu how they received a Service Unavailable
> error when they submitted a persona.  If I'm not mistaken, we're pretty sure
> it's the mail function timing out that causes these errors.  

What lead you to believe it was mail()?

Comment 5

[Jeremy Orem [:oremj]]

I've verified the php.ini is the same on all.

Comment 6

[Shae Rivard [:crimius]]

(In reply to comment #4)
> > An approver just sent me an email abotu how they received a Service Unavailable
> > error when they submitted a persona.  If I'm not mistaken, we're pretty sure
> > it's the mail function timing out that causes these errors.  
> 
> What lead you to believe it was mail()?

These errors have been going on for a long time, and when I brought it up to Ryan when I first started seeing them when approving personas, he said it was probably mail(), though I'm not sure why.

Comment 8

[Matthew N. [:MattN]]

I marked the original bug report (bug 598372) filed by rdhoherty as a dupe since this bug has more info.  That is where Ryan mentioned that PHP's mail() function may be involved.

(I'm also merging the priority & milestone fields)

Comment 9

[Wil Clouser [:clouserw]]

I don't think there are any mail() calls in a loop here.  Jeremy, can you tell us load on these boxes and SMTP traffic - it doesn't seem like there would enough mail to slow things down.

Also, can you configure it to use socketlabs to see if that will help.

Do the mail logs say anything about getting backed up or having trouble looking up domain names that would slow things down?

Comment 10

[Jeremy Orem [:oremj]]

Please attach the persona the submitter tried to upload when they received a timeout.

Comment 11

[Shae Rivard [:crimius]]

The submitter isn't sure which it was anymore, as they submitted several that day.  I'm getting service unavailable right now approving personas, on all of them I try to perform an action on, and after several tries they've finally gone through.  Two personas that gave this are <https://www.getpersonas.com/en-US/persona/333538> and <https://www.getpersonas.com/en-US/persona/333543>.

Comment 12

[Shae Rivard [:crimius]]

Another user reported this happening with this persona <http://www.getpersonas.com/en-US/persona/334208>

Comment 13

[Deb Richardson [:dria] (plz NEEDINFO)]

This is still happening, and it's not just when submitting Personas, but also when trying to review/approve stuff already in the queue.  Is there any way we can get the priority on this bumped up?  It's a major issue for the site and for our contributors.

Comment 14

[Matthew N. [:MattN]]

Attachment: Screenshot of POST request giving 'Service Unavailable'

While reviewing personas a few minutes ago, I received the 'Service Unavailable' error pages. When it happens, the request takes over 30 seconds to load and then other requests around that time tend to also get the same error.

The four errors I received for POST requests to https://www.getpersonas.com/en-US/admin/pending.php?category=Abstract

The response code was: HTTP/1.1 500 Internal Server Error
with the following headers:
Connection:close
Content-Type:text/html
Date:Tue, 11 Jan 2011 06:23:04 GMT

Perhaps there is something useful in the log files at this time? Hopefully the 30 second delay and the lack of response headers give clues as to where the problem is occurring?

Comment 15

[Jeremy Orem [:oremj]]

How often is this happening? Does submitting the form always take a long time?

Comment 16

[Matthew N. [:MattN]]

I'm not reviewing personas every day so I don't know how often it happens.  This form doesn't always take a long time as I just tested and it took around a second to submit.

Comment 17

[Deb Richardson [:dria] (plz NEEDINFO)]

It really seems to be intermittent.  I did a ton of reviews recently and had no problems at all.  I'll create a few new Personas in the next day or two and see what happens.

Comment 18

[Jeremy Orem [:oremj]]

(In reply to comment #17)
> It really seems to be intermittent.  I did a ton of reviews recently and had no
> problems at all.  I'll create a few new Personas in the next day or two and see
> what happens.

Have you run across this lately?

Comment 19

[Jeremy Orem [:oremj]]

This could have been due to storage load. We've reduced the load on that device and it sounds like it isn't happening very often. I'm closing this out.

Comment 20

[Matthew N. [:MattN]]

This is happening regularly again[1] and I suspect it's because of the high load due to the Firefox 4 release.

What can be done to prevent this problem? 

[1] https://forums.mozilla.org/addons/viewtopic.php?f=30&t=2675#p7647

Comment 21

[Matthew N. [:MattN]]

What is displaying this error page?  Is it something on the web head or is it the load balancer?  It doesn't seem to be a getpersonas.com error message.

Comment 22

[Wil Clouser [:clouserw]]

(In reply to comment #21)
> What is displaying this error page?  Is it something on the web head or is it
> the load balancer?  It doesn't seem to be a getpersonas.com error message.

It's the load balancer when the backend is unavailable (like the backend ran out of memory and didn't return a response or something).

Comment 23

[Jeremy Orem [:oremj]]

Please attach the POST data and full headers for a failed request?

Comment 24

[Jeremy Orem [:oremj]]

SMTP is still an issue here. If the submitters SMTP server is being slow or if the persona under review's users's SMTP server is slow the request will time out. Personas needs to stop using the mail() function directly.

Comment 25

[Matthew N. [:MattN]]

(In reply to comment #24)
> SMTP is still an issue here. If the submitters SMTP server is being slow or if
> the persona under review's users's SMTP server is slow the request will time
> out. Personas needs to stop using the mail() function directly.

Thanks for troubleshooting this

(In reply to comment #3)
> Possible solutions:
> - set up .ini config to use socketlabs explicitly
> - add configurations to personas and write wrappers for mail() that don't
> depend on server configs

Wil or Mike: Which of these two solutions is preferred and who can this be assigned to?

Comment 26

[Wil Clouser [:clouserw]]

Jeremy, can you just tell mail on that box to go through socketlabs?

Comment 27

[Jeremy Orem [:oremj]]

Those servers are using a smart-host now. Let's see if that helps.

Comment 28

[Matthew N. [:MattN]]

Attachment: POST /en-US/admin/pending.php (with request body)

(In reply to comment #27)
> Those servers are using a smart-host now. Let's see if that helps.

It doesn't seem to have helped for me or other reviewers[1].  Here are some headers of responses with the error message.

[1] https://forums.mozilla.org/addons/viewtopic.php?f=30&t=2599&start=75#p8356

Comment 29

[Matthew N. [:MattN]]

Attachment: GET /en-US/admin/dashboard.php

This one is a GET request

Comment 30

[Wil Clouser [:clouserw]]

getpersonas.com has been retired.  More information at https://blog.mozilla.org/addons/2013/04/11/background-themes-have-moved-to-amo/