563222 - Paypal is emailing our developers and disabling IPN on their accounts

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Wil Clouser [:clouserw]]

Attachment: Count of unique ids failing in logs

So...I actually had a blog post written for this, but it's not done yet so I'll write a summary here.  I'll emphasize right at the start though, this isn't affecting authors getting donations at all.

In normal PayPal workflow user1 receives funds from user2.  User1 is the merchant who runs the business, IPN server, etc.  If the IPN server goes down, paypal will email user1 and tell them it's not working.  Normally this works fine, however, in the case of AMO we're simply connecting user1 and user2, but specifying our own IPN server.  If our IPN server doesn't respond, user1 gets the email.  Not great, but I don't know of a better workflow here (-> add to agenda for paypal meeting? :)

I asked paypal what the threshold for sending emails is and they say it has to fail 100 times before they start emailing.  I looked in our logs (see attachment) and I dont see a count anywhere near that number.  Most of them are sitting around the mid-teens.

If paypal hits during downtime or something those requests won't be logged, but I wouldn't expect us to miss that many requests.  So, being PayPal, we're kind of shooting in the dark,

As to the point of this bug, the IPNs coming in aren't in our database.  I don't know if this is a code problem, a database problem, or a paypal problem.  If I had to guess I'd say someone sent an author money, we tried to save it to our db but it failed, and then sent the user to paypal (I realize this is concerning).

The patch I've written adds a little more logging but also adds a threshold of our own.  If an IPN coming in isn't identified on our end withing the first 10 tries (to allow for replication lag) we stop sending failures to paypal and instead send an OK.  This will prevent paypal from identifying our IPN server as broken, when, in fact, the IPN server is running fine, it just can't find the transaction paypal is talking about.

The case I'm writing the patch for should never happen, but it does appear to be an issue.  As we reduce load on the servers I expect it to disappear.

Overall affected users so far appears to be 36.  I don't know how many have gotten emails or had IPN disabled for their accounts but it is a non-zero number.  Additionally, I don't know how to reenable IPN for their accounts.  When the user follows PayPal's instructions to re-enable, PayPal asks for an IPN address for the account.  They shouldn't have to associate their account with us permanently, we send the IPN address along with the request.  If an account does have IPN disabled I don't know if that affects future reporting of donations to AMO or not, but it's certainly possible.  Another agenda item, I suppose.

tldr=paypal is still difficult to work with.

Comment 1

[Wil Clouser [:clouserw]]

jbalogh, r? http://github.com/clouserw/zamboni/commit/1c28eee8a3432c35871cf1254ed85b215726cdc8

Comment 2

[Wil Clouser [:clouserw]]

Alright, http://github.com/jbalogh/zamboni/commit/0f8b684b3cee0c5de7027ef17aa51039f5c5b319

Once this goes live, IPN shouldn't be disabled for people in the future and we'll have an accurate count of how many people it would have affected.

QA:  If you could run through 1 test and have it succeed I think we're alright.  This code should only affect transactions that have failed >10 times, which, I hope, is hard to reproduce.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

I donated $0.01 to https://preview.addons.mozilla.org/en-US/firefox/addon/3576/; not sure if I can really verify based on that, but at least we've tested it :-)

Comment 4

[KLB]

This bug has not been fixed.  I'm getting IPN failure notifications from Paypal again for donations sent via AMO.  Here is an example message I got this morning:
---begin message---
Please check your server that handles PayPal Instant Payment Notifications (IPN). IPNs
sent to the following URL(s) are failing:

/services/paypal
https://services.addons.mozilla.org/services/paypal

If you do not recognize this URL, you may be using a service provider that is using IPN on
your behalf. Please contact your service provider with the above information. If this
problem continues, IPNs may be disabled for your account. 

Thank you for your prompt attention to this issue.


Thanks,

PayPal 
---end message---

Comment 5

[Jeff Balogh (:jbalogh)]

(In reply to comment #4)
> This bug has not been fixed.  I'm getting IPN failure notifications from Paypal

Thanks for letting us know.  I filed bug 585065 to unblock that url.