903567 - Transaction not found when receiving Bango event notice

Status: RESOLVED FIXED

(Marketplace Graveyard :: Payments/Refunds, defect, P2)

Description

[Kumar McMillan [:kumar]]

Bango sends an event notice on purchase as of bug 866939. The XML notice contains a Bango transaction ID but solitude cannot look this up because it does not have a record of it at that time.

We need to fix this in the billing configuration step by posting a Bango transaction ID: https://github.com/mozilla/webpay/blob/master/lib/solitude/api.py#L200

The filter to look up the transaction may also need adjusting: https://github.com/mozilla/solitude/blob/master/lib/bango/forms.py#L385


Aug  9 12:36:24 localhost6.localdomain: [10.8.83.212] w.bango:ERROR Error calling solitude: Client Error 400: https://payments.allizom.org/bango/event/#012Content: {'notification': ['Transaction not found: 1231235685']}#012 :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20130809120512-2490723944/webpay/webpay/bango/views.py:171#012Traceback (most recent call last):#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20130809120512-2490723944/webpay/webpay/bango/views.py", line 168, in notification#012    'username': username#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/current/venv/lib/python2.6/site-packages/slumber/__init__.py", line 139, in post#012    resp = self._request("POST", data=s.dumps(data), params=kwargs)#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/current/venv/lib/python2.6/site-packages/curling/lib.py", line 216, in _request#012    content=self._try_to_serialize_error(resp))#012HttpClientError: Client Error 400: https://payments.allizom.org/bango/event/#012Content: {'notification': ['Transaction not found: 1231235685']}


Aug  9 12:36:24 localhost6.localdomain: [10.8.83.212] w.bango:DEBUG Bango notice: '\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><bangoEvents xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xml.bango.net/schemas/payment.xsd" version="1.0"><packageId>1013925</packageId><time>2013-08-09T19:36:22+00:00</time><source>PAYMENT</source><eventList><event><id>147153355</id><time>2013-08-09T19:14:40+00:00</time><source>PAYMENT</source><action>PAYMENT</action><data><item name="userId" value="1763337485" /><item name="status" value="OK" /><item name="bango" value="1113330000000774424" /><item name="paymentAmount" value="117" /><item name="paymentCurrency" value="USD" /><item name="paymentEarnings" value="67" /><item name="paymentSourceType" value="OPERATOR" /><item name="transId" value="1231235685" /></data></event></eventList></bangoEvents>' :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20130809120512-2490723944/webpay/webpay/bango/views.py:160

Comment 1

[Andy McKay]

There's multiple transaction ids floating around, I would do a sanity check that we are checking the correct one when we receive it from Bango (and that its filtered correctly).

Comment 2

[Andy McKay]

Hmm, i'm not sure. When is the transaction id used in the EventNotification generated? CreateBillingConfiguration returns us a billingConfigurationId, but not a TransactionID. We pass through an externalTransactionId to CreateBillingConfiguration, but that doesn't seem to get passed back in EventNotification.

Comment 3

[Andy McKay]

Oh I think we get the transaction id when the process has completed and Bango calls back to us:

https://github.com/mozilla/webpay/blob/master/webpay/bango/views.py#L49

But we are trying to have something that works for when that fails...

Comment 4

[Keir Kettle]

(In reply to Andy McKay [:andym] from comment #2)
> Hmm, i'm not sure. When is the transaction id used in the EventNotification
> generated? CreateBillingConfiguration returns us a billingConfigurationId,
> but not a TransactionID. We pass through an externalTransactionId to
> CreateBillingConfiguration, but that doesn't seem to get passed back in
> EventNotification.

Think you've probably answered this yourself, a transaction id is create when a transaction is completed. We return this in the querystring and will be in the event notification.

Comment 5

[Andy McKay]

Wondering how we can solve our original use case then, when the payment flow is interrupted and we never receive the transaction id. Is it possible to include external transaction id or billing configuration id in the event notification? Otherwise we have no way of matching it up.

Comment 6

[Kumar McMillan [:kumar]]

Ah, drat. This won't work unless we get the Mozilla transaction ID in the event notice. Keir, is it possible to send this in the event XML?

Comment 7

[Andy McKay]

Assigning to Kier, because I think this requires Bango changes.

Comment 8

[Kumar McMillan [:kumar]]

Keir, is there anything we can do to get a quick fix for this? In Durango, we've seen users double charged (bug 900899) because of various edge cases. We need server notifications to prevent double charging.

Comment 9

[Tom]

Will be live tomorrow.

Comment 10

[Keir Kettle]

Event notification XML will now include 'externalCPTransId' within the data element

Comment 11

[Andy McKay]

So far on stage I cannot see externalCPTransId. I can paste you some specific transactions if you'd like. Can I confirm this has been pushed to prod?

Comment 12

[Keir Kettle]

Investigating - Batches seem to be failing with (502) Bad Gateway?

Comment 13

[Andy McKay]

Currently we are throwing a 500 because we are expecting the element to be there. I'm going to fix it to return a 400. That might be manifesting as a 502.

Comment 14

[Tom]

no update currently. still investigating.

Comment 15

[Andy McKay]

Targetting for end of next week (1st Nov)

Comment 16

[Tom]

Update: We had a fix all ready for last week but when we started testing in LIVE we discovered that actually didn't solve the ultimate issue so we went back to drawing board and have created a new fix which will post the xml to Mozilla at time of payment which will solve multiple issues including bugs 894621, 902230, 903567. We will demonstrate this fix on the call tonight (Thursday) and will hopefully have this in LIVE next week following successful testing.

Comment 17

[Keir Kettle]

Workaround as discussed last week is now in production. If all OK please close this issue.

Comment 18

[Andy McKay]

We are getting the following in prod:

app.log-20131119.gz:Nov 18 21:25:39 localhost: [146.101.146.125] w.bango:INFO Bango notification encoding=None content_type=application/x-www-form-urlencoded POST keys=[u'XML'] :/data/mkt.prod/www/marketplace.firefox.com-webpay/deploy-webpay-prod-20131112193110-51734f3e6d/webpay/webpay/bango/views.py:165
app.log-20131119.gz:Nov 18 21:25:39 localhost: [146.101.146.125] w.bango:DEBUG Bango notice: '\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><bangoEvents xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xml.bango.net/schemas/payment.xsd" version="1.0"><packageId>1018155....

This has externalTransId in it, so raises an error.

We also get a notification that looks like this:

app.log-20131119.gz:Nov 18 20:54:06 localhost: [146.101.146.125] w.bango:INFO Bango notification received :/data/mkt.prod/www/marketplace.firefox.com-webpay/deploy-webpay-prod-20131112193110-51734f3e6d/webpay/webpay/bango/views.py:151
app.log-20131119.gz:Nov 18 20:54:06 localhost: [146.101.146.125] w.bango:INFO Bango notification encoding=None content_type=text/xml; encoding='utf-8' POST keys=[] :/data/mkt.prod/www/marketplace.firefox.com-webpay/deploy-webpay-prod-20131112193110-51734f3e6d/webpay/webpay/bango/views.py:165
app.log-20131119.gz:Nov 18 20:54:06 localhost: [146.101.146.125] w.cef:ERROR CEF Severity: 8 Message: MultiValueDictKeyError :/data/mkt.prod/www/marketplace.firefox.com-webpay/deploy-webpay-prod-20131112193110-51734f3e6d/webpay/webpay/base/utils.py:36

You said the format of the notification is exactly the same? According to the docs, the event notification is "for full HTTP compliance, the XML content is contained in a field called “XML” with
in the HTTP POST request, and the value is URL-encoded to make it web-safe". Odd that this other notification thats coming in has content type text/xml.

Comment 19

[Andy McKay]

Should we change change our accept end point to take text/xml and text/x-www-form-urlencoded or is this going to be the same as EventNotification?

Comment 20

[Tom]

We thought we had sent you the updated documentation to match our change. Apologies. It is a requirement for you to change on your side to accomplish a successful transaction checker.

Comment 21

[Andy McKay]

https://github.com/mozilla/webpay/pull/361

Updated, going to test on stage.

Comment 22

[Andy McKay]

There is still no externalCPTransId in there.

Dec  6 19:51:54 localhost6.localdomain: [10.8.83.212] w.bango:INFO Bango notification received :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20131205200417-8ba7172128/webpay/webpay/bango/views.py:151
Dec  6 19:51:54 localhost6.localdomain: [10.8.83.212] w.bango:INFO Bango notification encoding=None content_type=application/x-www-form-urlencoded POST keys=application/x-www-form-urlencoded :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20131205200417-8ba7172128/webpay/webpay/bango/views.py:166
Dec  6 19:51:54 localhost6.localdomain: [10.8.83.212] w.bango:DEBUG Bango notice: '\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><bangoEvents xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xml.bango.net/schemas/payment.xsd" version="1.0"><packageId>1019915</packageId><time>2013-12-06T19:51:53+00:00</time><source>PAYMENT</source><eventList><event><id>148340145</id><time>2013-12-06T19:45:14+00:00</time><source>PAYMENT</source><action>PAYMENT</action><data><item name="userId" value="1473894939" /><item name="status" value="OK" /><item name="bango" value="1113330000000803630" /><item name="paymentAmount" value="99" /><item name="paymentCurrency" value="USD" /><item name="paymentEarnings" value="69" /><item name="paymentSourceType" value="CARD" /><item name="transId" value="1338658095" /></data></event></eventList></bangoEvents>' :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20131205200417-8ba7172128/webpay/webpay/bango/views.py:177
Dec  6 19:51:54 localhost6.localdomain: [10.8.83.212] w.bango:ERROR Error calling solitude: Client Error 400: https://payments.allizom.org/bango/event/#012Content: {'notification': ['External trans id is required']}#012 :/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20131205200417-8ba7172128/webpay/webpay/bango/views.py:188#012Traceback (most recent call last):#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/deploy-webpay-stage-20131205200417-8ba7172128/webpay/webpay/bango/views.py", line 185, in notification#012    'username': username#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/current/venv/lib/python2.6/site-packages/curling/lib.py", line 160, in post#012    headers=headers, params=kwargs)#012  File "/data/addons-stage/www/marketplace.allizom.org-webpay/current/venv/lib/python2.6/site-packages/curling/lib.py", line 264, in _request#012    content=self._try_to_serialize_error(resp))#012HttpClientError: Client Error 400: https://payments.allizom.org/bango/event/#012Content: {'notification': ['External trans id is required']}

Comment 23

[Steve Ruston]

Please can you let us know if you can provide external transaction notice in event notice

Comment 24

[Keir Kettle]

As discussed back Nov - we created a moz specific solution, separate from our 'normal' event notification (so we could speed up the sending and allow more customisation etc).  The example above is from our normal event notification - as it looks like they are currently both switched on.  You can distinguish between the two based on contenttype - the 'normal' is application/x-www-form-urlencoded, the moz specific solution is text/xml. 

Can you confirm you are receiving two notifications & I will disable the original.

Comment 25

[Kumar McMillan [:kumar]]

Yep, I looked at today's log and I'm pretty sure we are getting dupes. There is always one notice that fails with 'No Authorization header' so that's probably the old one.

Comment 26

[Andy McKay]

If the later one isn't going to have the external transaction in it, then it might be worth turning off. The later transaction is nice though because it will repeat if there's a break down in server to server communication somewhere. Does the first one do that?

Comment 27

[Keir Kettle]

(In reply to Andy McKay [:andym] from comment #26)
Does the first one do that?

No it won't. 

Will switch off Event Notification for the time being.  Once we've completed the work required to update EV we can switch over so you've the benefit of a standard solution etc.

Comment 28

[Andy McKay]

(In reply to Keir Kettle from comment #2)
> Will switch off Event Notification for the time being.  Once we've completed
> the work required to update EV we can switch over so you've the benefit of a
> standard solution etc.

Ok, should we file a bug to track that?