901099 - Bango event notice is not valid XML

Status: RESOLVED FIXED

(Marketplace Graveyard :: Payments/Refunds, defect, P1)

Description

[Kumar McMillan [:kumar]]

Bango sends an event notice on purchase as of bug 866939. The notice is coming through and is authenticating (bug 898465) but it's an XML error:

Aug  2 20:39:10 <redacted>: [] s.bango:ERROR XML parse error :/data/solitude/www/payments.allizom.org/releases/1375473601-origin.mas/solitude/lib/bango/forms.py:344

Comment 1

[Kumar McMillan [:kumar]]

trying to get more info: https://github.com/mozilla/solitude/commit/74832e6ef5978c32554ba47749b49bbc3edbc308

Comment 2

[Kumar McMillan [:kumar]]

It's encoded when it reaches solitude:

Aug  2 22:57:09 ip-10-148-2-134.us-west-2.compute.internal: [] s.bango:DEBUG XML parse error from content: XML=%ef%bb%bf%3c%3fxml+version%3d%221.0%22+encoding%3d%22utf-8%22%3f%3e%3cbangoEvents+xmlns%3axsi%3d%22http%3a%2f%2fwww.w3.org%2f2001%2fXMLSchema-instance%22+xsi%3anoNamespaceSchemaLocation%3d%22http%3a%2f%2fxml.bango.net%2fschemas%2fpayment.xsd%22+version%3d%221.0%22%3e%3cpackageId%3e1013625%3c%2fpackageId%3e%3ctime%3e2013-08-02T22%3a57%3a11%2b00%3a00%3c%2ftime%3e%3csource%3ePAYMENT%3c%2fsource%3e%3ceventList%3e%3cevent%3e%3cid%3e147091405%3c%2fid%3e%3ctime%3e2013-08-02T22%3a55%3a46%2b00%3a00%3c%2ftime%3e%3csource%3ePAYMENT%3c%2fsource%3e%3caction%3ePAYMENT%3c%2faction%3e%3cdata%3e%3citem+name%3d%22userId%22+value%3d%221763270485%22+%2f%3e%3citem+name%3d%22status%22+value%3d%22OK%22+%2f%3e%3citem+name%3d%22bango%22+value%3d%221113330000000773337%22+%2f%3e%3citem+name%3d%22paymentAmount%22+value%3d%22117%22+%2f%3e%3citem+name%3d%22paymentCurrency%22+value%3d%22USD%22+%2f%3e%3citem+name%3d%22paymentEarnings%22+value%3d%2267%22+%2f%3e%3citem+name%3d%22paymentSourceType%22+value%3d%22OPERATOR%22+%2f%3e%3citem+name%3d%22transId%22+value%3d%221225365925%22+%2f%3e%3c%2fdata%3e%3c%2fevent%3e%3c%2feventList%3e%3c%2fbangoEvents%3e :/data/solitude/www/payments.allizom.org/releases/1375483409-origin.mas/solitude/lib/bango/forms.py:347

This should fix it:

>>> from urllib import unquote_plus
>>> unquote_plus('...')
'\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><bangoEvents xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xml.bango.net/schemas/payment.xsd" version="1.0"><packageId>1013625</packageId><time>2013-08-02T22:55:52+00:00</time><source>PAYMENT</source><eventList><event><id>147091405</id><time>2013-08-02T22:55:46+00:00</time><source>PAYMENT</source><action>PAYMENT</action><data><item name="userId" value="1763270485" /><item name="status" value="OK" /><item name="bango" value="1113330000000773337" /><item name="paymentAmount" value="117" /><item name="paymentCurrency" value="USD" /><item name="paymentEarnings" value="67" /><item name="paymentSourceType" value="OPERATOR" /><item name="transId" value="1225365925" /></data></event></eventList></bangoEvents>'

Comment 3

[Kumar McMillan [:kumar]]

There were a couple issues:
- Bango's request is somehow arriving in solitude in an encoded form. I added some logging to see if this is to do with the content-type header. I also added a hack to decode it in webpay first.
- Solitude was not parsing the XML correctly

Fixed:
https://github.com/mozilla/webpay/commit/f8cf139bc2223679b0e92935b9b8077454b2de7f
https://github.com/mozilla/solitude/commit/ba89087b0cd5f79206731d0ed554df097dd9a3fe

Comment 4

[Kumar McMillan [:kumar]]

Bango Test is not usable. Waiting on a stage push to test this...

Comment 5

[Kumar McMillan [:kumar]]

This is still causing a parse error in solitude. I learned that Bango's content-type is application/x-www-form-urlencoded but I can't write a test to reproduce it (yet)

Comment 6

[Kumar McMillan [:kumar]]

BOM isn't getting stripped (maybe), added some logging to see it better https://github.com/mozilla/solitude/commit/d9db9ed5221ab3ea5d1386c6fc29f9662e017d43

Comment 7

[Kumar McMillan [:kumar]]

the elusive BOM! https://github.com/mozilla/solitude/commit/c45211460f3e1e094188bc0628c8a8757ab09b06 I can't see that it's there in the log but it's not getting detected and stipped, trying to figure out why. bug 902230 is making it hard.

Comment 8

[Kumar McMillan [:kumar]]

Fixed: https://github.com/mozilla/webpay/commit/2490723944fb057c8a00c53c9bf3681a8060b957

Comment 9

[Kumar McMillan [:kumar]]

The XML is parsed right but now the transaction cannot be found: bug 903567 Getting closer...