Closed Bug 713777 Opened 12 years ago Closed 3 years ago

Amazon modifies and re-signs apk after uploading [was "Cannot install Firefox from Amazon appstore if Firefox Beta already exists on device"]

Categories

(Firefox for Android Graveyard :: General, defect, P5)

ARM
Android

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: tchung, Unassigned)

References

Details

Attachments

(4 files)

This is likely a releng bug, so redirect to correct component if so.

If Firefox beta pre-exists on your device (downloaded from Android Market), installing Firefox from Amazon Appstore will fail and throw a cert error:

12-27 16:49:22.560: DEBUG/PackageManager(2696): Scanning package org.mozilla.firefox
12-27 16:49:22.560: DEBUG/PackageManager(2696): Shared UserID org.mozilla.firefox.sharedID (uid=10105): packages=[PackageSetting{40671e58 org.mozilla.firefox_beta/10105}]
12-27 16:49:22.560: ERROR/PackageManager(2696): Package org.mozilla.firefox has no signatures that match those in shared user org.mozilla.firefox.sharedID; ignoring!
12-27 16:49:22.560: WARN/PackageManager(2696): Package couldn't be installed in /data/app/org.mozilla.firefox-1.apk


Screenshot and logcat attached

Repro
1) Samsung Galaxy S 2 (android 2.3.3) and Galaxy Nexus (Android 4.0.1)
2) successfully install Firefox Beta (v10b1) from Android Market
3) launch Amazon Appstore, search for Firefox (v9), and install when found
4) Verify installation process goes through the steps, and will fail on installation. 

Expected:
- ability to coexist firefox beta and firefox from Amazon

Actual: 
- cert error halts Firefox Amazon installation
Attached file logcat
Summary: Cannot install Firefox from Amazon appstore of Firefox Beta already exists on device → Cannot install Firefox from Amazon appstore if Firefox Beta already exists on device
Component: General → Release Engineering
Product: Fennec Native → mozilla.org
QA Contact: general → release
Version: Trunk → other
I grabbed the apk that the amazon app store pulled down (vnz72847.apk), the android market installed (org.mozilla.firefox-1.apk) and the one we posted to ftp (fennec-9-multi.apk). The one from amazon is bigger.

$ ls -l *.apk
-rw-r--r--  1 blassey  staff  15426132 27 Dec 20:42 fennec-9-multi.apk
-rw-r--r--  1 blassey  staff  15426132 27 Dec 20:36 org.mozilla.firefox-1.apk
-rw-r--r--  1 blassey  staff  15484445 27 Dec 20:37 vnz72847.apk


So, Alex... is Amazon re-packing our apk? If they are I don't think that's really acceptable and we should consider pulling it from the app store.
(tip of hat to lsblakk for helping here)


We have confirmed that something/someone has modified these bits in our fennec apk after we generated them and uploaded them to Amazon.


How we got the apk from amazon & checked signatures:
1. install the amazon appstore app
2. search for & install Firefox (installation itself failed)
3. attach USB and go to /Volumes/NO NAME/Android/data/com.amazon.venezia/cache/vnz1931126841.apk (downloaded apk name seems to be randomized each time since two attempts to install the same Firefox app results in two different names in the cache dir: vnz1931126841.apk and vnz-840065667.apk)
4. on a build slave:
    export JAVA_HOME=/builds/jdk
    export PATH=/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
5. run http://hg.mozilla.org/build/tools/file/234d651b3a15/release/signing/verify-android-signature.sh as:
./verify-android-signature.sh --apk=http://avnerd.tv/sharedFiles/Apps/vnz-840065667.apk --tools-dir=../.. -r
(and then run that again with http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/9.0/android/multi/fennec-9.0.multi.android-arm.apk )

Differences we can see so far:
1) In the .apk 
ftp.m.o:
369667 Fri Dec 16 15:39:02 PST 2011 META-INF/MANIFEST.MF
369788 Fri Dec 16 15:39:02 PST 2011 META-INF/RELEASE.SF
  1451 Fri Dec 16 15:39:02 PST 2011 META-INF/RELEASE.RSA
amazon:
369817 Sat Dec 24 00:49:38 PST 2011 META-INF/MANIFEST.MF
369938 Sat Dec 24 00:49:40 PST 2011 META-INF/APKSIGNE.SF
  1117 Sat Dec 24 00:49:40 PST 2011 META-INF/APKSIGNE.RSA
amazon also adds to the package
see here:
[cltbld@moz2-linux-slave51 signing]$ tail output.txt (jar tvf of amazon apk)
   356 Sat Dec 24 00:49:36 PST 2011 application.ini
   134 Sat Dec 24 00:49:36 PST 2011 platform.ini
 90610 Sat Dec 24 00:49:36 PST 2011 greprefs.js
   153 Sat Dec 24 00:49:36 PST 2011 chrome.manifest
     6 Fri Dec 16 14:05:18 PST 2011 update.locale
    54 Fri Dec 16 14:05:18 PST 2011 removed-files
 13216 Sat Dec 24 00:49:36 PST 2011 plugin-container
  4160 Sat Dec 24 00:49:28 PST 2011 kiwi
264188 Sat Dec 24 00:49:32 PST 2011 classes.dex  <-- MUCH LARGER IN AMAZON APK!
    19 Sat Dec 24 00:49:32 PST 2011 com.amazon.content.id.MC-S-11UR8SLI9K7UN <-- NEW!
[cltbld@moz2-linux-slave51 signing]$ jar tvf fennec-9.0.multi.android-arm.apk | tail
   356 Fri Dec 16 14:06:04 PST 2011 application.ini
   134 Fri Dec 16 14:06:04 PST 2011 platform.ini
 90610 Fri Dec 16 14:06:04 PST 2011 greprefs.js
   153 Fri Dec 16 14:06:04 PST 2011 chrome.manifest
     6 Fri Dec 16 14:05:18 PST 2011 update.locale
    54 Fri Dec 16 14:05:18 PST 2011 removed-files
 13216 Fri Dec 16 14:06:04 PST 2011 plugin-container
116932 Fri Dec 16 14:05:20 PST 2011 classes.dex


2) The timestamp of the files within the apk have changed.
ftp.m.o: Fri Dec 16 15:39:02 PST 2011
amazon: Sat Dec 24 00:49:32 PST 2011


3) As blassey noted in comment#2, the file size of the apk is different. 



Given that:
* We dont know what/who has changed these files (we suspect amazon has modified after we uploaded, but need to verify this with Amazon.)
* We dont know how users will be impacted by this change
...therefore, I recommend we disable this app in amazon  store until we debug this issue. Legneato has approved this plan on phone. 

NOTE: The Amazon store doesnt let you immediately pull an app, you can only say "please stop offering it after a date in the future", so I've marked Fennec 9.0 to not be offered starting 28dec2011 at 00:00, and filed a support question with amazon about making this happen sooner if possible.
From https://developer.amazon.com/help/faq.html, I see:

....
Can I apply DRM to my app?
For each app that you submit to the Amazon Appstore, you can choose to apply DRM or make your app available without any rights management constraints. If you do choose to apply DRM to one of your apps, you must use the DRM system provided by Amazon through the Amazon Appstore Developer Portal.
[NOTE: we posted app with no DRM]
.....
Can I apply a signature to my app?
All applications must be digitally signed with a certificate. The default signature applied to your app is a certificate supplied by Amazon that is unique to your developer account. If your signing strategy requires that a different certificate be applied, you may do so by submitting a request via the "Questions about application signatures" subject in the Contact Us section of the Amazon Appstore Developer Portal. Please indicate the title of the application for which you are submitting the request.
[:akeybl, when you posted the app to the store, did you do this?]

.....
How does Amazon prepare my binary for the Appstore?
Amazon wraps your binary with code that allows the Amazon Appstore to collect health and stability analytics related to the app, evaluate and enforce our program policies, and share aggregated information with you and others regarding the program.
....
(In reply to John O'Duinn [:joduinn] from comment #4)
> Can I apply a signature to my app?
> All applications must be digitally signed with a certificate. The default
> signature applied to your app is a certificate supplied by Amazon that is
> unique to your developer account. If your signing strategy requires that a
> different certificate be applied, you may do so by submitting a request via
> the "Questions about application signatures" subject in the Contact Us
> section of the Amazon Appstore Developer Portal. Please indicate the title
> of the application for which you are submitting the request.
> [:akeybl, when you posted the app to the store, did you do this?]

We did not contact Amazon separately - we only requested that they not DRM the application through the normal interface. The possibility of this signing issue was not caught prior to submission.

I understand the decision made to pull from the Amazon Appstore given the possible update risk. Out of curiosity though, do we believe that this is only limited to phones where Beta was installed previously, or are we concerned that this may also impact our update strategy to FF10 (through the Amazon Appstore)? Note that we're not using our in-app updater, and we're only updating through the Amazon Appstore. We're also targeting Kindle Fire users for the large majority, so this may be a more secondary concern if Kindle Fire updates through the Amazon Appstore are left unaffected.
Brad has clarified that our concern is not limited to updating, but that there is also concern with what unknowns Amazon may have added to the APK for the purposes of "collect[ing] health and stability analytics related to the app".
Assignee: nobody → joduinn
Severity: normal → major
I'm more concerned with "evaluate and enforce our program policies" quite frankly.
per irc: 

1) legneato is main contact with amazon; single point of communications should help avoid confusion, so pushing this bug to him.
2) akeybl had filed amazon case#49543161 for same issue. Legneato will 


If it turns out that Amazon's modification of our uploaded apk is non-negotiable, we'll need to start investigation of:
* is this policy a blocker to us being in amazon app store?
* what does this amazon code do (any mozilla-policy violations?)
* what does this amazon code mean for testing? updates?
Assignee: joduinn → clegnitto
Summary: Cannot install Firefox from Amazon appstore if Firefox Beta already exists on device → Amazon modifies and re-signs apk after uploading [was "Cannot install Firefox from Amazon appstore if Firefox Beta already exists on device"]
(In reply to John O'Duinn [:joduinn] from comment #8)
> per irc: 
> 
> 1) legneato is main contact with amazon; single point of communications
> should help avoid confusion, so pushing this bug to him.
> 2) akeybl had filed amazon case#49543161 for same issue. Legneato will 
> 
> 
> If it turns out that Amazon's modification of our uploaded apk is
> non-negotiable, we'll need to start investigation of:
> * is this policy a blocker to us being in amazon app store?
> * what does this amazon code do (any mozilla-policy violations?)
> * what does this amazon code mean for testing? updates?

I'm particularly interested in the updates and testing piece.   we've never had a clear path to testing which market would handle updates, Android Market or Amazon Appstore?
On my mind tonight; is there any chance those users who downloaded 9 off Amazon (few and far between, right?) are stranded for future updates?
(In reply to Aaron Train [:aaronmt] from comment #10)
> On my mind tonight; is there any chance those users who downloaded 9 off
> Amazon (few and far between, right?) are stranded for future updates?

Speaking of updates, the amazon appstore app offered me an update to Firefox.  (i preinstalled firefox 9 from Appstore before we had them pull it)

Can someone inform amazon to stop issuing updates until we put Firefox back?
Attached image screenshot 1
update page screenshot 1
screenshot offering the update
(In reply to Tony Chung [:tchung] from comment #11)
> (In reply to Aaron Train [:aaronmt] from comment #10)
> > On my mind tonight; is there any chance those users who downloaded 9 off
> > Amazon (few and far between, right?) are stranded for future updates?
> 
> Speaking of updates, the amazon appstore app offered me an update to
> Firefox.  (i preinstalled firefox 9 from Appstore before we had them pull it)
> 
> Can someone inform amazon to stop issuing updates until we put Firefox back?

That's...strange. We had a conf call with them scheduled for today that got moved to a later date. I'll see what they can do.
(In reply to Tony Chung [:tchung] from comment #13)
> Created attachment 585468 [details]
> amazon update screenshot 2
> 
> screenshot offering the update

I confirmed that with FF9 installed through the Android Market that no updates were offered to me through the Appstore. Did you install FF8 from the Android Market prior to being offered the update?
(In reply to Alex Keybl [:akeybl] from comment #15)
> (In reply to Tony Chung [:tchung] from comment #13)
> > Created attachment 585468 [details]
> > amazon update screenshot 2
> > 
> > screenshot offering the update
> 
> I confirmed that with FF9 installed through the Android Market that no
> updates were offered to me through the Appstore. Did you install FF8 from
> the Android Market prior to being offered the update?

I recall having installed FF9 Android Market at one point in time, uninstalling, and re-installing FF9 Amazon Appstore.  My current setup still has FF9 Amazon Appstore on my device, and has triggered these updates (build date: 20111216).
fyi, on my android LG g2x phone, I did the following just now: 

0) on the amazon.com app store, I verified that Firefox is not found when I search for it.

1) deleted firefox9.0 and amazon marketplace app from my phone, and rebooted my phone.

2) went to amazon.com on my phone, and installed their amazon.com app store app.

3) logged into the amazon.com app using my amazon account usr/pswd.

4) the amazon app immediately prompted me to re-install Firefox9.0. It then downloaded FF9, and then prompted me to accept privs needed. Once I accepted privs, I then get an error "Application not installed". 

This looks like some form of caching, but should be investigated in discussions with amazon.
Was there an outcome to this? Or are we still talking with Amazon?
Still talking with Amazon
Have they confirmed how they fiddled with our bits or are we still waiting for that?
Update: BD is handling outreach. They are contacting Amazon ~ 1-2 weeks. No news, yet.
Assignee: LegNeato → release-mgmt
Component: Release Engineering → General
Product: mozilla.org → Firefox
Version: other → unspecified
Product: Firefox → Firefox for Android
I was thinking about this. Could we just use a different App ID org.mozilla.firefox_amazon? With different app names users would not be able to get into this corrupted state.
(In reply to Kevin Brosnan [:kbrosnan] from comment #22)
> I was thinking about this. Could we just use a different App ID
> org.mozilla.firefox_amazon? With different app names users would not be able
> to get into this corrupted state.

It's really the implications of signing/changing the APK without a request from Mozilla, as opposed to the instance where you install the APK from elsewhere and then install the APK from the Appstore.
Was this ever resolved? Firefox seems to run great on a Kindle Fire HD 8.9", but I have to manually install from the apk download.
Blocks: kindle
I'm also interested in seeing Fennec run on the Kindle Fire HDX, since that became accessible with FireOS 3.0 and higher, and the stock browser isn't accessible, except the web views that are used in Amazon's own shopping apps.
Marco: you might be interested in tracking Bug 956964.
Lukas, can you check that?
Flags: needinfo?(lsblakk)
I'm going to have to check in with John about this as he might have been the only person with the account access we can use to test this.  Will leave the ni? and report back when I know more.
Mark now has account access to the Amazon store so passing this over to him for further investigation/confirmation that we can use their system without apk issues.
Flags: needinfo?(lsblakk)
Assignee: release-mgmt → mark.finkle
Is there any update on this? If the concern is over "code purity" or support level, could one conceivably release a less-supported unbadged build of fennec to the Amazon App Store to bridge the gap?
Flags: needinfo?(mark.finkle)
The APK is still modified without the DRM checkbox.

A small amount of code is added to do logging and crash reporting ("Kiwi"), and system integration ("Venezia").

Still assessing.
Flags: needinfo?(mark.finkle)
More findings, as best I can determine from decompilation:

* The APK is re-signed with a different key. The key uses strings that presumably come from the Amazon Appstore account: "Release Engineering Mozilla", not those used to build the APK.

  * The owner of the key -- Amazon -- can ship apps to Android devices that have access to private Fennec data sources, thanks to Android's signature permission capability.

  * Apps signed with another key -- e.g., other apps that Mozilla delivers outside of the Amazon Appstore -- don't have signature permissions to Mozilla apps delivered via the Appstore, because the keys differ.

  * I have read that Amazon supports dual-signing, but this is cumbersome (upload unsigned, download the munged APK, sign it, upload it again for signing) and possibly not supported correctly by Android. More investigation needed.

* Files are added to the APK: 

 Name: com.amazon.content.id.MC-S-H9M0F8QF5YUQ
 SHA1-Digest: Y0zfatMHSiIM7DFh22Rh6A/lHqM=

 Name: com.amazon.kiwi.version
 SHA1-Digest: FcgJQag08Uqa6Bp2QDfEl0veK7c=

 Name: kiwi
 SHA1-Digest: sbRaTtgNc8hCzLn2gTiq/qbffUA=

* Classes are added to classes.dex.

* The manifest is rebuilt (as a result of re-jarsigning). Hashes aren't changed for image resources. Preference XML files etc. *do* seem to be rebuilt (at least, their hashes change). I don't know what changes are applied.

* The builds I have don't allow me to tell if .so files are modified. Modifying these libraries would be a huge red flag, so another cycle should be performed to test that.

* Activity classes are modified to intercept every create/pause/delete/result:

  public void onCreate(Bundle paramBundle)
  {
    onCreateBrowserApp(paramBundle);
    Kiwi.onCreate(this, false);
  }

  public Dialog onCreateDialog(int paramInt)
  {
    Dialog localDialog = Kiwi.onCreateDialog(this, paramInt);
    if (localDialog != null)
      return localDialog;
    return super.onCreateDialog(paramInt);
  }

* The Kiwi class logs, and potentially intercepts data. For example, it receives and processes every activity result:

  public static boolean onActivityResult(Activity paramActivity, int paramInt1, int paramInt2, Intent paramIntent)
  {
    if (preProcess("onActivityResult", paramActivity))
    {
      com.amazon.android.e.f localf = new com.amazon.android.e.f(paramActivity, paramInt1, paramInt2, paramIntent);
      return INSTANCE.resultManager.a(localf);
    }
    return false;
  }
Do we know if Amazon still repack the apk?
I don't know. It would be hard to check without someone uploading our app to the Amazon App Store…
The mobile team used to create minimal test APKs to upload and verify on the various app stores.
Karen Rudnitski was in active discussion with Amazon in 2015 about their resigning. It was not something they were willing to change. From https://developer.amazon.com/public/support/submitting-your-app/tech-docs/submitting-your-app

> On submission, Amazon wraps your app with additional code that enables the app to communicate with the Amazon Appstore client to collect analytics, evaluate and enforce our program policies, share aggregated information with you and others regarding the program, and for other purposes. You can use Live App Testing to see how this additional code will impact the behavior of your app.
Setting to P5 because there's nothing we can do right now.
Priority: -- → P5
Assignee: mark.finkle → nobody
Has this been resolved now that there's Firefox for Fire TV?

https://www.amazon.com/dp/B078B5YMPD
No. That is a Android WebView browser.
So Mozilla is perfectly fine with shipping a browser using the system-provided Android WebView (version 55 on Fire OS; more than a year old!) that gets modified by Amazon when uploaded to their store but not fine with shipping a browser with a current Gecko version included that gets modified by Amazon when uploaded to their store?
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.