1397303 - Test out Google Play's disposable signing keys (aka Google Play App Signing)

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

During the last Google IO, Google released a new feature: We can rotate APKs signing keys. Before then, we could only have one and only one signing key during the whole lifetime of an APK. This is an OS restriction. Android doesn't accept a certificate of an installed app to change.

With this new feature, Google Play will be able to re-sign APKs for us. This means, the long lived certificate will be store on Google Play and we only store an signing key that Google Play will verify. This means, Android won't see any change of signature. On our end, however, we'll be able to rotate keys at will[1].

This feature sounds cool at first, but there may be some risks. Some questions we have in mind are:
* How can we rotate a disposable signing key? 
* What happens if we loose the disposable signing key? Are we still able to rotate it?
* Can we still upload an APK with the real signing key?
* Can we opt out from disposable signing keys?
* Are incorrectly signed APKs still rejected, even when they were signed with the disposable key? (See bug 1313995 for a test plan)
* Can we upload APKs - signed with the real or disposable key - onto others App Stores (like Yandex or Amazon)? 
* Are we allowed to download fully signed APKs from Google Play, so we can store them elsewhere (archive.mozilla.org, for instance)? 


[1] https://support.google.com/googleplay/android-developer/answer/7384423?hl=en

Comment 1

[Johan Lorenzo [:jlorenzo]]

Some questions have answers in the documentation:

* How can we rotate a disposable signing key? 
* What happens if we loose the disposable signing key? Are we still able to rotate it?
> Lost or compromised private keys
> 
> If you're enrolled in Google Play App Signing, you can reset your
> upload key if:
> 
>     You lost your private key, or
>     Your private key has been compromised
> 
> Note: Resetting your upload key will not affect the app signing key
> that Google Play uses to re-sign APKs before delivering to users.
> Reset your upload key
> Step 1: Generate a new private key and upload certificate
> 
> To generate and register a new upload key, follow the instructions
> in the Android Studio Help Center. The new key must be different than
> your previous key.
> 
> Then, export the certificate for the new key to PEM format:
> 
> keytool -export -rfc -alias <upload> -file <upload_certificate.pem> -keystore <keystore.jks>
> 
> 
> Step 2: Contact our support team
> 
> Our support team only accepts key reset requests from the Play
> Console account owner.
> 
> To contact us, the account owner can fill out this form. Make sure
> to attach the upload_certificate.pem file.
> 
> You'll receive an email once we've registered the new upload key.
> At that time, you can follow the steps listed above to update your
> keystores and API provider registration.


* Can we opt out from disposable signing keys?
> Important: App Signing opt-in is permanent
> 
> Google Play App Signing is an optional program. If you prefer, you can
> continue managing your own keys.
> 
> Once you've enrolled your app in Google Play App Signing, withdrawal is
> not supported. To preserve the security of your app signing keys, we don't
> have the ability to remove keys from the secure server.

Comment 2

[Johan Lorenzo [:jlorenzo]]

* Can we still upload an APK with the real signing key?
No, error message gotten:
> Upload failed
> You uploaded an APK that is not signed with the upload certificate. You must use the same certificate. The upload certificate has fingerprint:
> [ SHA1: E6:48:C9:42:D7:AC:03:7C:FD:33:84:A6:6B:FB:93:C5:4F:9F:3B:11 ]
> and the certificate used to sign the APK you uploaded have fingerprint:
> [ SHA1: 97:95:D4:7E:40:F9:8C:2F:10:71:11:2E:63:A0:11:9F:B6:7B:4E:8D ]



* Are incorrectly signed APKs still rejected, even when they were signed with the disposable key? (See bug 1313995 for a test plan)
Yes. All uploads we done via the web interface.
APK with no signature:
> Upload failed
> You uploaded an APK with an invalid signature (learn more about signing). Error from apksigner: ERROR: JAR_SIG_NO_MANIFEST: Missing META-INF/MANIFEST.MF

Deleted file in APK (thus, missing manifest references a non-existing file):
> Upload failed
> You uploaded an APK with an invalid signature (learn more about signing). Error from apksigner: ERROR: JAR_SIG_MISSING_ZIP_ENTRY_REFERENCED_IN_MANIFEST: res/layout/tooltip.xml entry referenced by META-INF/MANIFEST.MF not found in the APK

File changed in APK (checksum in manifest isn't good anymore):
> Upload failed
> You uploaded an APK with an invalid signature (learn more about signing). Error from apksigner: ERROR: JAR_SIG_ZIP_ENTRY_DIGEST_DID_NOT_VERIFY: SHA-256 digest of res/layout/tooltip.xml does not match the digest specified in META-INF/MANIFEST.MF. Expected: <Q2k3kIlgPQfy+2bU8cxGkPT/Yh+CnlA7lGKAOaPmhoY=>, actual: <47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=>

New file in APK (non-existing entry in manifest)
> Upload failed
> You uploaded an APK with an invalid signature (learn more about signing). Error from apksigner: ERROR: JAR_SIG_NO_ZIP_ENTRY_DIGEST_IN_MANIFEST: No digest for alien_file in META-INF/MANIFEST.MF



* Can we upload APKs - signed with the real or disposable key - onto others App Stores (like Yandex or Amazon)?
- Disposable key: Not recommended. APKs aren't modified by other app stores. This would means we shipped one App signed with 2 different keys. Android doesn't support 2 different keys for the same APK. This would prevent users from switching from a store to another.
- Real key, but signed by Google: Looks doable technically. More details in the next question.


* Are we allowed to download fully signed APKs from Google Play, so we can store them elsewhere (archive.mozilla.org, for instance)?
Yes, but only by clicking a button on the admin web interface. The account should have admin privileges (read-only permissions didn't seem to work). There is currently no way to download an APK with the developper API[1].

Google Play does change the content of APKs. Thanks to diffoscope[2], I was able to compare:
- an APK downloaded from Google Play before using disposable keys
- another APK after that feature was turned on.

I noted:
- Different version code. This is expected. Google Play requires you to upgrade the version code at each submission.
- Some files had their zip header changed from there are an "extended local header and an ''extra field'' associated with the file" to only the "extra-field". The zipinfo man page [3] is not very explicit about the meaning of the "extra-field".
- For each zip headers changed, the manifest had a new checksum.
- META-INF/ICE.{SF,RSA} became META-INF/GOOGPLAY.{SF,RSA}. These files are named after the certificate alias in the keystore. There is no way to rename the one used by Google Play.
- Last but not least: AndroidManifest.xml got an XML element:
> <meta-data android:name="com.android.vending.derived.apk.id" android:value="1"/>

It's currently documented at [4]:
> Apps that are signed by Google will have a “derived APK ID” written into their AndroidManifest.xml file. You'll see a meta-data element added under the application tag that references <meta-data android:name="com.android.vending.derived.apk.id" android:value="[ID]" />.
>
> This ID is the identifier of the modified APK and will be reported in the usual bug reporting tools. You can use the derived APK ID to recognize a specific APK that was delivered by Play.
>
> To download the Google Play signed APK in your Play Console, go to Release management > Artifact library.

I'm not sure how the ID is calculated. My guess is: they increment it at each product we flip "Google Play App signing" on.




NEW QUESTIONS
* Can a user manually install an APK signed by the real key and then manually update via the APK signed by Google.
Yes. I managed to do so on Andoid 6.0.1.



* How can we make sure Google Play doesn't modify APKs too much?
Like said above, diffoscope makes a good job a showing the internal and structural differences of APKs. It analyzes the structure of the ZIP archive: how many files are present, what their metadata, what the size of the archives are, what's the compression ratio. It decodes the binary manifest and show their differences. It diffs every file in the archive, unless the file is new or deleted.

However, these diffs are clobbered by 2 things:
- the zip headers of 50+ (I haven't counted) files are changed, before Google Play and after.
- because the changed zip headers, the manifest has its checksums recalculated.

As a consequence, looking if a file is missing or has been added requires to process the whole file list and manifest, unless we generate Zip archives like Google Play does.



* Can we automate this verification?
In theory, yes. Here's what we should do:
1. Crate a script that dowloads the APK from Google Play. Sadly, a webdriver script will have to click on the UI. Such scripts are known to be flaky. I haven't found an HTTP request that we may be able to craft. Knowing the "artifact_id" (seems to be incremented each time any App developed pushes an APK), and the authentication token are the biggest hurdles.
2. Find the right Google Play permissions to give to that script.
3. Find the same zip parameters as Google Play uses. Implement them in either the build step or a repackage step.
4. Create a script that uses diffoscope to verify:
- META-INF/KEY_ALIAS.{SF,RSA} are replaced by META-INF/GOOGPLAY.{SF,RSA}
- the new signature can be verified by the known public key
- Only AndroidManifest.xml has its SHA-256-Digest changed
- AndroidManifest.xml got only the "derived" XML element added




[1] https://developers.google.com/android-publisher/api-ref/edits/apks
[2] https://diffoscope.org/
[3] http://www.info-zip.org/mans/zipinfo.html
[4] https://support.google.com/googleplay/android-developer/answer/7384423 See "Modification of your APK".

Comment 3

[Johan Lorenzo [:jlorenzo]]

To sum up: 

+---------------------------------------------------------------------------------------------------------------------+
|                          Pros                           |                           Cons                            |
+---------------------------------------------------------|-----------------------------------------------------------+
| * Google signs APK for us, so we may not need a heavy   | * A signing infrastructure remains required. Rotating key |
|   signing infrastructure anymore.                       |   requires a human's action on Google Play's end          |
|                                                         |                                                           |
| * APKs signed by Google can be downloaded to be reused  | * APKs signed by Google are explicitly marked. Users can  |
|   elsewhere.                                            |   see "GOOGPLAY.RSA" by just unzipping the archive. This  |
|                                                         |   may concern users who want to stay out of the Google    |
|                                                         |   ecosystem, by downloading APKs manually or on           |
|                                                         |   alternative app stores, for instance. Nonetheless, as   |
|                                                         |   of today, we aren't aware of any alternative use of     |
|                                                         |   this marker.                                            |                 
|                                                         |                                                           |
| * Users can upgrade to GP's signed APKs without         | * If we want to double-check what comes from Google Play, |
|   technical difficulties.                                 |   then we require some additional investigation and       |
|                                                         |   tweaking. Steps are listed in the last answer above.    |
|                                                         |                                                           |
|                                                         | * Once Google Play App Signing is on, there is no turning |
|                                                         |   back.                                                   |
+---------------------------------------------------------------------------------------------------------------------+

I'm going to close this bug, because every known question got answered. If you have any more, feel free to reopen/NI me.

Comment 4

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)
> - For each zip header changed, the manifest had a new checksum.

That is incorrect (thanks :catlee for spotting it!). The checksum itself isn't changed. Checksums are calculated against the original file alone (and not its position in the zip archive). Nonetheless, because of the headers changed, the order of the files in the manifest *is* modified. That's what makes the diff of the manifests hard to parse.

Comment 5

[Hal Wine [:hwine] use NI!]

(In reply to Johan Lorenzo [:jlorenzo] - On PTO - Back on July 19th from comment #4)
> ... the order of the files in the manifest *is* modified. That's what
> makes the diff of the manifests hard to parse.

Just in case this gets implemented - are both manifests sorted -- but perhaps using collation rules from different locales? 

If so, it might be worth changing our locale to match googles. I've seen this most often with the difference between the 'C' and 'en_US-UTF8' collating sequences.

Comment 6

[Johan Lorenzo [:jlorenzo]]

If I remember correctly, these manifest are sorted. I'll check this out. Worst case scenario, the test script can still read both manifest and store them in dictionaries, and then we can check both dictionaries are equal.

Comment 7

[Johan Lorenzo [:jlorenzo]]

If I remember correctly, these manifest aren't sorted. I'll check this out. Worst case scenario, the test script can still read both manifest and store them in dictionaries, and then we can check both dictionaries are equal.