(Synth APK) - APK Factory fails on Twitter webapp manifest

RESOLVED FIXED

Status

()

Firefox for Android
Web Apps
P1
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: aaronmt, Unassigned)

Tracking

29 Branch
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [A4A])

(Reporter)

Description

4 years ago
$ curl http://dapk.net/application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp

TypeError: Cannot read property 'length' of undefined

* Adding handle: conn: 0x7f8a62803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f8a62803a00) send_pipe: 1, recv_pipe: 0
* About to connect() to dapk.net port 80 (#0)
*   Trying 50.16.201.4...
* Connected to dapk.net (50.16.201.4) port 80 (#0)
> GET /application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp HTTP/1.1
> User-Agent: curl/7.30.0
> Host: dapk.net
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< X-Powered-By: Express
< Content-Type: text/plain
< Content-Length: 53
< Date: Tue, 28 Jan 2014 16:57:56 GMT
< Connection: keep-alive
< 
* Connection #0 to host dapk.net left intact
I *think* this is the BOM bug, which I believe Kumar was looking at, although I can't find the bug number now.  cc:ing Kumar and ozten.

Comment 2

4 years ago
I'll take a look as the BOM bug #960618 was deployed mid-day yesterday.

Comment 3

4 years ago
ERROR CONTROLLR: Error downloading http://mobile.twitter.com/cache/twitter.webapp

ERROR CONTROLLR: TypeError: Object #<Object> has no method 'forEach'
    at module.exports (/home/ozten/apk-factory-service/lib/owa_validator.js:49:25)
    at /home/ozten/apk-factory-service/lib/owa_downloader.js:34:21
    at Request._callback (/home/ozten/apk-factory-service/lib/file_loader.js:54:11)
    at Request.self.callback (/home/ozten/apk-factory-service/node_modules/request/index.js:148:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/home/ozten/apk-factory-service/node_modules/request/index.js:891:14)
    at Request.EventEmitter.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/home/ozten/apk-factory-service/node_modules/request/index.js:842:12)
    at IncomingMessage.EventEmitter.emit (events.js:117:20)
    at _stream_readable.js:920:16

Updated

4 years ago
QA Contact: aaron.train → ozten.bugs

Comment 4

4 years ago
Fixed in https://github.com/mozilla/apk-factory-service/commit/9072075bb1972f1fe7f0bab6f5c712f3e1f0aa1c

However... our AWS is being rate limited by Twitter. This could be because another VM outside of our control is a Twitter based app or ???

Error when curling from dapk.net

curl -v 'http://mobile.twitter.com/cache/twitter.webapp'
* About to connect() to mobile.twitter.com port 80 (#0)
*   Trying 199.16.156.43... connected
> GET /cache/twitter.webapp HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: mobile.twitter.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< cache-control: max-age=86400
< content-type: text/plain; charset=utf-8
< date: Tue, 28 Jan 2014 17:54:27 GMT
< expires: Wed, 29 Jan 2014 17:54:27 GMT
< retry-after: 3600
< server: tfe
< set-cookie: _mobile_sess=BAh7BjoLcmVwX2lkSSIpMjc4YWY1MjAtNmE3My0wMTMxLWZkM2EtMDA4Y2Zh%0AMDQyZjU4BjoGRVQ%3D%0A--a91c55e107b7b3caad58cec2b1877fa0008b6d70; path=/; expires=Sun, 16-Mar-2014 03:12:45 GMT
< set-cookie: guest_id=v1%3A139093166760151839; Domain=.twitter.com; Path=/; Expires=Thu, 28-Jan-2016 17:54:27 UTC
< status: 403 Forbidden
< vary: Accept-Encoding
< Transfer-Encoding: chunked
<
Forbidden (Rate Limit Exceeded)

Comment 5

4 years ago
Tightened up error handling and redeployed.

The server now responds with 'Manifest is Forbidden CODE: 403'

Chatting with digitarald who is helped make the Twitter OWA.

Comment 6

4 years ago
CCing digitarald - any feedback from Twitter folks?
Duplicate of this bug: 965567
Jeremy, could you please check with your infrastructure team if they can remove the rate limiting from the manifest.webapp file? The manifest should never return a 403. Also rate limiting doesn't seem to make sense on a /cache/ folder that holds static assets.
Flags: needinfo?(jforrester)

Comment 9

4 years ago
Cross posted to the web team to see what's going on - i'll let you know when I hear more
Flags: needinfo?(jforrester)

Comment 10

4 years ago
Can we get the mozilla's ip ranges you're requesting from, we should be able to up the rate limits for you by IP
Short term - here is the IP
50.16.201.4

Problems
1) This is our shared dev environment (not stage or production which will have the same issue)

2) We're deployed on AWS, so this will change

3) This system that exposed the problem is only one of many potential clients that would contact Twitter for the OWA manifest. Those other clients could also fail silently.

I think this is too fragile. Do we have a medium term solution?

thanks for the quick response and all your help!

Updated

4 years ago
Blocks: 958329

Comment 12

4 years ago
Quick question To be rate limited you've got to be sending us a ton of requests from the same IP address. 

Can you explain a little what the service aims to do (I got the general gist from reading the APK factory wiki pages) and why you're making so many requests to the web manifest file?
Thanks Jeremy!

> Quick question To be rate limited you've got to be sending us a ton of requests from the same IP address. 

The first request I made from this AWS box was rate limited.

I was shocked that we are getting rate limited, as I don't think we are sending you very man requests. Could others on AWS be contributing to our rate?

Do you have an logs that show the frequency of requests to this url?

> Can you explain a little what the service aims to do (I got the general gist from reading the APK factory > wiki pages) and why you're making so many requests to the web manifest file?

TLDR; We shouldn't be sending much traffic from our public development environment. Production will use HTTP caching sensibly.

Once we are in production... The first time someone requests the Twitter open web app (OWA) for Firefox on Android a request will come in to our service. Our service will retrieve the OWA manifest from Twitter and build an Android native app. Building the app takes about 5 - 10 seconds. If additional requests come in for the Twitter app, they should block in a build queue and not cause more traffic to Twitter.
This .apk installer file is put into a file cache and is served out to the client. Subsequent requests for the Twitter app will check to see if the OWA manifest url at Twitter's contents have changed using HTTP caching. 

We don't have our stage or production environments up which will use Squid as a reverse HTTP proxy cache. This would use HEAD requests and TTL to reduce traffic to Twitter's manifest url.

In development, we currently do a get request for the OWA manifest URL for each build request. This SHOULD NOT be happening more than a couple times an hour, as people are only doing manual testing.

If the Twitter manifest changes, than this invalidates the cache and we'll re-build the Android App.

Keep in mind, we are in development, so we might clear caches and make more requests.

Comment 14

4 years ago
Are you still able to recreate the issue btw?
Yes

curl 'http://dapk.net/application.apk?manifestUrl=http://mobile.twitter.com/cache/twitter.webapp' > twitter.apk

Actual
cat twitter.apk
Manifest is Forbidden CODE: 403

Expected
file twitter.apk
twitter.apk: Zip archive data, at least v2.0 to extract

Updated

4 years ago
Flags: needinfo?(jforrester)
jforrester - any updates? Any way we can get /cache/ urls removed from your rate limiting system?
We're headed into Mobile World Congress and Twitter doesn't work from the Marketplace.

Workaround - Sideload http://ozten.com/random/apk-factory/twitter.apk

Updated

4 years ago
Whiteboard: A4A

Updated

4 years ago
Priority: -- → P1

Updated

4 years ago
Whiteboard: A4A → [A4A]
(Reporter)

Comment 18

4 years ago
This is still an issue. Note mozilla-29 will be beta the week of the 18th.
(Reporter)

Comment 19

4 years ago
I randomly tested Twitter off the live Marketplace this morning and was able to successfully install. This seems resolved. Can we get confirmation of any changes Twitter changed?
This is great news!

The only problem with closing this bug w/o explicit notification that Twitter changed something, is that it can happen again.

Without a fix, it means that when you made the request, we didn't get rate-limited. That doesn't mean we're not rate-limited from the stage box, nor that we won't get rate-limited in the future.

prod release https://controller.apk.firefox.com 
prod review https://controller-review.apk.firefox.com
stage https://apk-controller.stage.mozaws.net

all appear to be working.

So... I'll close, but explicit confirmation from our friends at Twitter would be awesome.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED

Updated

4 years ago
Blocks: 984032
Resolution: FIXED → WORKSFORME

Comment 21

4 years ago
Still trying to hunt down if this was fixed by a code change on our side - it's not immediately apparent it was (i.e. no one indicated on our jira that this issue was fixed) 

Let me know if it comes up again
Flags: needinfo?(jforrester)
Just a note, Twitter is broken for reviewers, since we always fetch the manifest when doing review builds and this bug is not fixed.

Comment 23

4 years ago
This should be fixed now - let me know if you still see any issues

Comment 24

3 years ago
Are we fine to close this issue? Austin are you still able to reproduce this when doing review builds?
I'm sorry I missed responding to Comment#23.

Yes, this still reproduces.

curl -v -k https://controller-review.apk.firefox.com/application.apk?manifestUrl=https://mobile.twitter.com/cache/twitter.webapp -o twitter.apk

Actual: http status 400, response is {}

Expected: http status 200, response is an APK file. Running `file twitter.apk` would report a Zip archive format.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

Comment 26

3 years ago
Hey Austin, can you email me directly jforrester at twitter - I want to put you in contact directly with my engineering team to resolve this 

(In reply to Austin King [:ozten] from comment #25)
> I'm sorry I missed responding to Comment#23.
> 
> Yes, this still reproduces.
> 
> curl -v -k
> https://controller-review.apk.firefox.com/application.apk?manifestUrl=https:/
> /mobile.twitter.com/cache/twitter.webapp -o twitter.apk
> 
> Actual: http status 400, response is {}
> 
> Expected: http status 200, response is an APK file. Running `file
> twitter.apk` would report a Zip archive format.

Updated

3 years ago
Depends on: 1019384
The Twitter team has deployed a fix, looks great!
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.