838856 - Speed up app purchases on the phone

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Wifi, defect)

Description

[krupa raj[:krupa]]

I usually try to do all my testing on Mozilla and Mozilla-G.

Making an app purchase is excruciatingly slow and for today, i havent been able to make a single payment.

Our users might try to make app purchases on crappy wifis and we need to deal with it by a)speeding up the whole process b) showing appropriate user messages/feedback when the connection is too slow.

The user experience of purchasing an app is very painful right now.


top offenders:
Duration:       39.478 42bd58d0 ***** (marketplace-dev.allizom.org -> POST /timing/record)
Duration:       43.833 49ac6310 ***** (marketplace-dev.allizom.org -> )
Duration:       20.583 49ac5620 ***** (marketplace-dev.allizom.org -> POST /services/csp/report)
Duration:       50.766 49ac5060 ***** (native-persona.org -> GET /include.js)
Duration:       35.663 49ac54b0 ***** (marketplace-dev-cdn.allizom.org -> GET /mozpay/jsi18n.js)

Comment 1

[Andy McKay]

Reporting on CSP is an issue, we shouldn't be doing that at all because there should be no CSP problems.

The timing is nice to have, but we can worry about that later, we can slow that down too.

Let's remove jsi18n.js for now.

Comment 2

[Andy McKay]

native-persona.org/include.js has an e-tag on it, but i'm not sure whats responsible for caching that.

Comment 3

[Andy McKay]

Google analytics. 

CSP WARN:  Directive img-src https://marketplace-dev.allizom.org:443 http://dev2.addons.phx1.mozilla.com:80 https://www.google.com:443 https://www.getpersonas.com:443 https://s3.amazonaws.com:443 https://marketplace-dev-cdn.allizom.org:443 violated by https://ssl.google-analytics.com/__utm.gif

CSP WARN:  Directive img-src https://marketplace-dev.allizom.org:443 http://dev2.addons.phx1.mozilla.com:80 https://www.google.com:443 https://www.getpersonas.com:443 https://s3.amazonaws.com:443 https://marketplace-dev-cdn.allizom.org:443 violated by https://ssl.google-analytics.com/__utm.gif

Comment 4

[Wil Clouser [:clouserw]]

If Andy hijacked this bug for CSP, fine, but I witnessed this today on Krupa's phone and I think it's a platform bug with, I'm guessing, some kind of backoff/retry algorithm.

You can see from her logs it affects multiple sites including Identity, Marketplace, and the Marketplace CDN.  It's not reproducible consistently.  It only happens on the device (not on the computer).  You can see the network traffic icon on the phone disappear for long periods of time while you're waiting for the page to load.

My uneducated guess is that the connection fails for some reason early on and the device decides to retry but backs off for a certain amount of time before it does.  Maybe several times.  This would explain all the symptoms here but to fully test that theory we'd need someone doing testing in a controlled environment (probably through a proxy).  I'm CCing Dietrich to comment on the idea.  Dietrich - does an algorithm like that exist in the device?  Any other reasons this would be happening?  I can't explain it taking 35 seconds to reach our CDN from the marketplace context.

Comment 5

[Andy McKay]

I've cleaned up the CSP, removed the jsi18n request and reduced the amount of timings. I can't speed the rest up however.

Krupa: if you see any CSP posts on prod, please file a bug, we shouldn't be having any when using the marketplace.

Comment 6

[Jason Smith [:jsmith]]

Adding a couple of people here who might have some insight on the above discussion.

My initial guesses is that if this is a platform bug, it would either be:

1. Some sort of lame network bug
2. A regression from landing some of the work with captive portal

Comment 7

[Jason Smith [:jsmith]]

I'm going to operate under the assumption right now this is a platform bug, so let's send this into triage to see what triage thinks.

Comment 8

[Dietrich Ayala (:dietrich)]

Are all these connections being made from the Marketplace client app on the device? How is the connection being made? XHR? There's no "auto retry" magic at that layer of the stack. It would be in your application level code, afaict.

Comment 9

[Jason Duell]

> I think it's a platform bug with, I'm guessing, some kind of backoff/retry algorithm.
> ...I can't explain it taking 35 seconds to reach our CDN from the marketplace context.

ccing :mcmanus in case he can think of anything here.  But this smells like something higher up than necko to me.  Other than things like queuing requests if there are too many sockets open, we generally rely on TCP to handle our backoff logic.  (If you lose the initial SYN packets for a TCP connection, TCP itself does a poor job of promptly reissuing them, but to work around that we open 2 TCP connections, so you'd have to lose both SYN packets: possible if your network has a lot of packet loss, but then you'd be seeing *all* network traffic sucking--not just app purchases).

> 2. A regression from landing some of the work with captive portal

Could be--I haven't followed that work closely.

It would be useful here to get one or both of

1) an NSPR log for the HTTP traffic (this will also capture XHR):
 
   https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging#Firefox_OS_phones

2) A wireshark capture of the network traffic, as a pcap file.

Comment 10

[Jason Duell]

Is this happening for other folks, or just :krupa?  If the latter than lame-network is more of a possibility.  What's the packet loss rate on Mozilla-G wifi?

Comment 11

[Wil Clouser [:clouserw]]

(In reply to Jason Duell (:jduell) from comment #9)
> 1) an NSPR log for the HTTP traffic (this will also capture XHR):
>  
>   
> https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/
> HTTP_logging#Firefox_OS_phones


I think this is what I saw the original numbers on.  Krupa - do you have that log?

Comment 12

[Wil Clouser [:clouserw]]

(In reply to Jason Duell (:jduell) from comment #10)
> Is this happening for other folks, or just :krupa?  If the latter than
> lame-network is more of a possibility.  What's the packet loss rate on
> Mozilla-G wifi?

Yeah, it's worth asking if this is on all connections or only when buying an app.  Krupa?

Comment 13

[krupa raj[:krupa]]

(In reply to Wil Clouser [:clouserw] from comment #12)
> (In reply to Jason Duell (:jduell) from comment #10)
> > Is this happening for other folks, or just :krupa?  If the latter than
> > lame-network is more of a possibility.  What's the packet loss rate on
> > Mozilla-G wifi?
> 
> Yeah, it's worth asking if this is on all connections or only when buying an
> app.  Krupa?

Payments are very slow. Other interactions with marketplace are pretty reasonable.

Comment 14

[Alex Keybl [:akeybl]]

(In reply to krupa raj 82[:krupa] from comment #13)
> (In reply to Wil Clouser [:clouserw] from comment #12)
> > (In reply to Jason Duell (:jduell) from comment #10)
> > > Is this happening for other folks, or just :krupa?  If the latter than
> > > lame-network is more of a possibility.  What's the packet loss rate on
> > > Mozilla-G wifi?
> > 
> > Yeah, it's worth asking if this is on all connections or only when buying an
> > app.  Krupa?
> 
> Payments are very slow. Other interactions with marketplace are pretty
> reasonable.

Comment 0 suggests that we were not able to make any payments - what speed are you seeing now?

Comment 15

[Jason Smith [:jsmith]]

I think this may be another instance of bug 840612.

Comment 16

[Jason Smith [:jsmith]]

(In reply to krupa raj 82[:krupa] from comment #13)
> (In reply to Wil Clouser [:clouserw] from comment #12)
> > (In reply to Jason Duell (:jduell) from comment #10)
> > > Is this happening for other folks, or just :krupa?  If the latter than
> > > lame-network is more of a possibility.  What's the packet loss rate on
> > > Mozilla-G wifi?
> > 
> > Yeah, it's worth asking if this is on all connections or only when buying an
> > app.  Krupa?
> 
> Payments are very slow. Other interactions with marketplace are pretty
> reasonable.

Just to link up the other discussions in different bugs - the people that seem to reproduce this issue all seem to work from the Mt View office and can reproduce there. Blake is coming to Mt View tomorrow to meet with Stephen, myself, and Krupa.

I think we're all seeing the same wifi bug in weird strange ways.

Comment 17

[Jason Smith [:jsmith]]

And now we know they are different bugs. Krupa - Can you still reproduce this?

Comment 18

[Andrew Overholt [:overholt]]

Is this still reproducible?

Comment 19

[krupa raj[:krupa]]

All the GET requests associated with the purchases I have made today are ~3 secs. 

However, Some of the people who were trying to make payments from Spain were complaining about how slow the entire flow is. I will keep the bug open to get their feedback.

Comment 20

[Jason Smith [:jsmith]]

My gut feeling on this bug is if there is a perf bug going on in the payments process, we probably want to start a meta bug on this with investigation (and find someone to investigate with a fine-comb). Then, file bugs for the major perf issues causing the process to be slow.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Not tracking since this seems like a meta bug for various pieces that need to come together (perf assessment, marketplace decisions).

Comment 22

[Hubert Figuiere [:hub]]

Hard to test without bug 855143

Comment 23

[Mike Lee [:mlee]]

Bill, is this still a concern for Markteplace?

Comment 24

[Bill Maggs (bmaggs)]

I don't have the updated sense of the team for how the speed of the payments flow, especially with a lot of new markets coming on line. Krupa, what is the latest?

Comment 25

[krupa raj[:krupa]]

The slowness we are seeing are mostly on Persona pages and Bango pages. I think we can close this bug and address those issues in different bugs.

Comment 26

[Hubert Figuiere [:hub]]

Please provide the bug numbers for the individual problems so we can track as well.

Comment 27

[Mike Lee [:mlee]]

Krupa, please provide bug numbers for the issues you mentioned in comment 25.