972462 - Japanese characters are replaced with ??? on Bango purchase screen

Status: RESOLVED FIXED

(Marketplace Graveyard :: Payments/Refunds, defect, P1)

Description

[krupa raj[:krupa]]

Attachment: special_chars_3.jpg

steps to reproduce:
1. Try to purchase the paid app named "Test App (おすすめの拡張機能 )"
2. Sign in
3. Enter PIN
4. On the Confirm your Purchase screen, notice the app name

actual behavior:
app name is replaced with ????. See screenshot.

Comment 1

[Andy McKay]

First lets check that we are passing the right thing to Bango.

Comment 2

[Kumar McMillan [:kumar]]

This is where we pass the name to Bango: https://github.com/mozilla/webpay/blob/master/webpay/pay/tasks.py#L194-L195

Comment 3

[Jared Lockhart [:jlockhart]]

It seems as though we're sending the SOAP XML to Bango in UTF-8 encoding.  Perhaps on their end they're decoding that incorrectly or storing it in a different character encoding.

Example SOAP XML being sent in UTF-8:

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="com.bango.webservices.billingconfiguration" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <ns0:Body>
      <ns1:CreateBillingConfiguration>
         <ns1:request>
            <ns1:username>user</ns1:username>
            <ns1:password>password</ns1:password>
            <ns1:bango>bango_id</ns1:bango>
            <ns1:typeFilter>
               <ns1:string>OPERATOR</ns1:string>
               <ns1:string>PSMS</ns1:string>
               <ns1:string>INTERNET</ns1:string>
               <ns1:string>CARD</ns1:string>
            </ns1:typeFilter>
            <ns1:priceList>
               <ns1:Price>
                  <ns1:amount>1</ns1:amount>
                  <ns1:currency>USD</ns1:currency>
               </ns1:Price>
            </ns1:priceList>
            <ns1:externalTransactionId>transaction_id</ns1:externalTransactionId>
            <ns1:pageTitle>japanese: おすすめの拡張機能</ns1:pageTitle>
            <ns1:configurationOptions>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>APPLICATION_CATEGORY_ID</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>18</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>BILLING_CONFIGURATION_TIME_OUT</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>1860</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>REDIRECT_URL_ONERROR</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>https://marketplace-dev.allizom.org/mozpay/err</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>REQUEST_SIGNATURE</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>abc</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>REDIRECT_URL_ONSUCCESS</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>https://marketplace-dev.allizom.org/mozpay/ok</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>MOZ_USER_ID</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>user_id</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
               <ns1:BillingConfigurationOption>
                  <ns1:configurationOptionName>APPLICATION_SIZE_KB</ns1:configurationOptionName>
                  <ns1:configurationOptionValue>1</ns1:configurationOptionValue>
               </ns1:BillingConfigurationOption>
            </ns1:configurationOptions>
         </ns1:request>
      </ns1:CreateBillingConfiguration>
   </ns0:Body>
</SOAP-ENV:Envelope>

Comment 4

[Keir Kettle]

We do not have full unicode support in our data storage, and in our experience, neither do many MNO billing systems.

Comment 5

[Wil Clouser [:clouserw]]

(In reply to Keir Kettle from comment #4)
> We do not have full unicode support in our data storage, and in our
> experience, neither do many MNO billing systems.

What's your recommendation?

Comment 6

[Andy McKay]

Would be nice to know what Bango supports so we can show messages to developers, Keir is looking into this for us.

Comment 7

[Andy McKay]

Bango will update their backend to support to UTF8.

Comment 8

[Keir Kettle]

Update - meeting with systems / dev is scheduled for 10/4.  Should be able to timings update after that meeting.

Comment 9

[Keir Kettle]

Discussed with systems / dba guys and basically this is a db issue - the columns that content titles encodes as ISO-8859-1, asp.net etc use UTF-16. They are investigating the impact of altering the column.  Will update once I've a date for making the switch.

Comment 10

[Steve Ruston]

Keir can you update on the target for this being completed

Comment 11

[Keir Kettle]

Aiming for available on prod 22/5

Comment 12

[Keir Kettle]

We need to do additional impact analysis on changes - this means we are now aiming to have available on prod 29/5.  Apologies for any inconvenience.

Comment 13

[Keir Kettle]

Hi we've hit some problems deploying this to our production database. Planning to push this now on Tuesday 3/6 - apologies for the inconvenience.

Comment 14

[Andy McKay]

This is on payments-alt and should be tested there. If its good we'll promote up through stage to prod.

Comment 15

[Madalin Cotetiu]

Giving Bug 1008343 is there a workaround to test an actual payment on payments-alt?

Comment 16

[Andy McKay]

As a hosted app.