964845 - (Synth APK) - APK Factory fails on Twitter webapp manifest

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Web Apps (PWAs), defect, P1)

Description

[Aaron Train [:aaronmt]]

$ curl http://dapk.net/application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp

TypeError: Cannot read property 'length' of undefined

* Adding handle: conn: 0x7f8a62803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f8a62803a00) send_pipe: 1, recv_pipe: 0
* About to connect() to dapk.net port 80 (#0)
*   Trying 50.16.201.4...
* Connected to dapk.net (50.16.201.4) port 80 (#0)
> GET /application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp HTTP/1.1
> User-Agent: curl/7.30.0
> Host: dapk.net
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< X-Powered-By: Express
< Content-Type: text/plain
< Content-Length: 53
< Date: Tue, 28 Jan 2014 16:57:56 GMT
< Connection: keep-alive
< 
* Connection #0 to host dapk.net left intact

Comment 1

[Myk Melez [:myk] [@mykmelez]]

I *think* this is the BOM bug, which I believe Kumar was looking at, although I can't find the bug number now.  cc:ing Kumar and ozten.

Comment 2

[Austin King [:ozten]]

I'll take a look as the BOM bug #960618 was deployed mid-day yesterday.

Comment 3

[Austin King [:ozten]]

ERROR CONTROLLR: Error downloading http://mobile.twitter.com/cache/twitter.webapp

ERROR CONTROLLR: TypeError: Object #<Object> has no method 'forEach'
    at module.exports (/home/ozten/apk-factory-service/lib/owa_validator.js:49:25)
    at /home/ozten/apk-factory-service/lib/owa_downloader.js:34:21
    at Request._callback (/home/ozten/apk-factory-service/lib/file_loader.js:54:11)
    at Request.self.callback (/home/ozten/apk-factory-service/node_modules/request/index.js:148:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/home/ozten/apk-factory-service/node_modules/request/index.js:891:14)
    at Request.EventEmitter.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/home/ozten/apk-factory-service/node_modules/request/index.js:842:12)
    at IncomingMessage.EventEmitter.emit (events.js:117:20)
    at _stream_readable.js:920:16

Comment 4

[Austin King [:ozten]]

Fixed in https://github.com/mozilla/apk-factory-service/commit/9072075bb1972f1fe7f0bab6f5c712f3e1f0aa1c

However... our AWS is being rate limited by Twitter. This could be because another VM outside of our control is a Twitter based app or ???

Error when curling from dapk.net

curl -v 'http://mobile.twitter.com/cache/twitter.webapp'
* About to connect() to mobile.twitter.com port 80 (#0)
*   Trying 199.16.156.43... connected
> GET /cache/twitter.webapp HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: mobile.twitter.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< cache-control: max-age=86400
< content-type: text/plain; charset=utf-8
< date: Tue, 28 Jan 2014 17:54:27 GMT
< expires: Wed, 29 Jan 2014 17:54:27 GMT
< retry-after: 3600
< server: tfe
< set-cookie: _mobile_sess=BAh7BjoLcmVwX2lkSSIpMjc4YWY1MjAtNmE3My0wMTMxLWZkM2EtMDA4Y2Zh%0AMDQyZjU4BjoGRVQ%3D%0A--a91c55e107b7b3caad58cec2b1877fa0008b6d70; path=/; expires=Sun, 16-Mar-2014 03:12:45 GMT
< set-cookie: guest_id=v1%3A139093166760151839; Domain=.twitter.com; Path=/; Expires=Thu, 28-Jan-2016 17:54:27 UTC
< status: 403 Forbidden
< vary: Accept-Encoding
< Transfer-Encoding: chunked
<
Forbidden (Rate Limit Exceeded)

Comment 5

[Austin King [:ozten]]

Tightened up error handling and redeployed.

The server now responds with 'Manifest is Forbidden CODE: 403'

Chatting with digitarald who is helped make the Twitter OWA.

Comment 6

[Austin King [:ozten]]

CCing digitarald - any feedback from Twitter folks?

Comment 8

[:Harald Kirschner :digitarald]

Jeremy, could you please check with your infrastructure team if they can remove the rate limiting from the manifest.webapp file? The manifest should never return a 403. Also rate limiting doesn't seem to make sense on a /cache/ folder that holds static assets.

Comment 9

[Jeremy Forrester]

Cross posted to the web team to see what's going on - i'll let you know when I hear more

Comment 10

[Jeremy Forrester]

Can we get the mozilla's ip ranges you're requesting from, we should be able to up the rate limits for you by IP

Comment 11

[Austin King [:ozten]]

Short term - here is the IP
50.16.201.4

Problems
1) This is our shared dev environment (not stage or production which will have the same issue)

2) We're deployed on AWS, so this will change

3) This system that exposed the problem is only one of many potential clients that would contact Twitter for the OWA manifest. Those other clients could also fail silently.

I think this is too fragile. Do we have a medium term solution?

thanks for the quick response and all your help!

Comment 12

[Jeremy Forrester]

Quick question To be rate limited you've got to be sending us a ton of requests from the same IP address. 

Can you explain a little what the service aims to do (I got the general gist from reading the APK factory wiki pages) and why you're making so many requests to the web manifest file?

Comment 13

[Austin King [:ozten]]

Thanks Jeremy!

> Quick question To be rate limited you've got to be sending us a ton of requests from the same IP address. 

The first request I made from this AWS box was rate limited.

I was shocked that we are getting rate limited, as I don't think we are sending you very man requests. Could others on AWS be contributing to our rate?

Do you have an logs that show the frequency of requests to this url?

> Can you explain a little what the service aims to do (I got the general gist from reading the APK factory > wiki pages) and why you're making so many requests to the web manifest file?

TLDR; We shouldn't be sending much traffic from our public development environment. Production will use HTTP caching sensibly.

Once we are in production... The first time someone requests the Twitter open web app (OWA) for Firefox on Android a request will come in to our service. Our service will retrieve the OWA manifest from Twitter and build an Android native app. Building the app takes about 5 - 10 seconds. If additional requests come in for the Twitter app, they should block in a build queue and not cause more traffic to Twitter.
This .apk installer file is put into a file cache and is served out to the client. Subsequent requests for the Twitter app will check to see if the OWA manifest url at Twitter's contents have changed using HTTP caching. 

We don't have our stage or production environments up which will use Squid as a reverse HTTP proxy cache. This would use HEAD requests and TTL to reduce traffic to Twitter's manifest url.

In development, we currently do a get request for the OWA manifest URL for each build request. This SHOULD NOT be happening more than a couple times an hour, as people are only doing manual testing.

If the Twitter manifest changes, than this invalidates the cache and we'll re-build the Android App.

Keep in mind, we are in development, so we might clear caches and make more requests.

Comment 14

[Jeremy Forrester]

Are you still able to recreate the issue btw?

Comment 15

[Austin King [:ozten]]

Yes

curl 'http://dapk.net/application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp' > twitter.apk

Actual
cat twitter.apk
Manifest is Forbidden CODE: 403

Expected
file twitter.apk
twitter.apk: Zip archive data, at least v2.0 to extract

Comment 16

[Austin King [:ozten]]

jforrester - any updates? Any way we can get /cache/ urls removed from your rate limiting system?

Comment 17

[Austin King [:ozten]]

We're headed into Mobile World Congress and Twitter doesn't work from the Marketplace.

Workaround - Sideload http://ozten.com/random/apk-factory/twitter.apk

Comment 18

[Aaron Train [:aaronmt]]

This is still an issue. Note mozilla-29 will be beta the week of the 18th.

Comment 19

[Aaron Train [:aaronmt]]

I randomly tested Twitter off the live Marketplace this morning and was able to successfully install. This seems resolved. Can we get confirmation of any changes Twitter changed?

Comment 20

[Austin King [:ozten]]

This is great news!

The only problem with closing this bug w/o explicit notification that Twitter changed something, is that it can happen again.

Without a fix, it means that when you made the request, we didn't get rate-limited. That doesn't mean we're not rate-limited from the stage box, nor that we won't get rate-limited in the future.

prod release https://controller.apk.firefox.com 
prod review https://controller-review.apk.firefox.com
stage https://apk-controller.stage.mozaws.net

all appear to be working.

So... I'll close, but explicit confirmation from our friends at Twitter would be awesome.

Comment 21

[Jeremy Forrester]

Still trying to hunt down if this was fixed by a code change on our side - it's not immediately apparent it was (i.e. no one indicated on our jira that this issue was fixed) 

Let me know if it comes up again

Comment 22

[Austin King [:ozten]]

Just a note, Twitter is broken for reviewers, since we always fetch the manifest when doing review builds and this bug is not fixed.

Comment 23

[Jeremy Forrester]

This should be fixed now - let me know if you still see any issues

Comment 24

[Jeremy Forrester]

Are we fine to close this issue? Austin are you still able to reproduce this when doing review builds?

Comment 25

[Austin King [:ozten]]

I'm sorry I missed responding to Comment#23.

Yes, this still reproduces.

curl -v -k https://controller-review.apk.firefox.com/application.apk?manifestUrl=https://mobile.twitter.com/cache/twitter.webapp -o twitter.apk

Actual: http status 400, response is {}

Expected: http status 200, response is an APK file. Running `file twitter.apk` would report a Zip archive format.

Comment 26

[Jeremy Forrester]

Hey Austin, can you email me directly jforrester at twitter - I want to put you in contact directly with my engineering team to resolve this 

(In reply to Austin King [:ozten] from comment #25)
> I'm sorry I missed responding to Comment#23.
> 
> Yes, this still reproduces.
> 
> curl -v -k
> https://controller-review.apk.firefox.com/application.apk?manifestUrl=https:/
> /mobile.twitter.com/cache/twitter.webapp -o twitter.apk
> 
> Actual: http status 400, response is {}
> 
> Expected: http status 200, response is an APK file. Running `file
> twitter.apk` would report a Zip archive format.

Comment 27

[Austin King [:ozten]]

The Twitter team has deployed a fix, looks great!