705807 - Android signing-on-demand

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Aki Sasaki (not active)]

https://bugzilla.mozilla.org/show_bug.cgi?id=705214#c1

As noted in that bug, not producing nightly-signed bits will break tegra testing.
I *think* we can get around this, but it might not be possible due to Android permissions.  We need to verify that only producing release-signed bits will work from a testing perspective, or generate both nightly- and release-signed bits before we can turn this on.

Comment 1

[Aki Sasaki (not active)]

Multilocale will not be going away (Amazon Market), so most likely our requirements for an Android release build will be

a) nightly-signed en-US apk
b) release-signed en-US apk
c) nightly-signed multi apk
d) release-signed multi apk

Comment 2

[Aki Sasaki (not active)]

We may as well expand this for nightlies, so we have a defined and similar way to sign for both nightlies and releases.

Comment 3

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #0)
> https://bugzilla.mozilla.org/show_bug.cgi?id=705214#c1
> 
> As noted in that bug, not producing nightly-signed bits will break tegra
> testing.

I'm going to revisit a lot of this while porting remote talos/unittests to mozharness.
I think I'm going to re-sign every apk with a test-specific key before installing, which will allow us to create only nightly signed bits during nightly jobs and only release signed bits during release jobs.

(In reply to Aki Sasaki [:aki] from comment #1)
> Multilocale will not be going away (Amazon Market)

Multilocale is now only enabled on android-xul.
When we shift entirely to android native, there will be no more multilocale.

I think we're unblocked here from these two perspectives.

Comment 4

[[e-mail] (:bhearsum)]

Marc volunteered to have a look at this if he has time.

Comment 5

[Alex Keybl [:akeybl]]

Since daily mobile betas are not a part of the rapid betas plan, I don't believe this is a blocker as noted on [1].

[1] https://wiki.mozilla.org/Release_Management/Rapid_Betas#RelEng.2FIT

Comment 6

[[e-mail] (:bhearsum)]

We've been talking about the possibility of building Android as part of rapid/automated betas for a couple of reasons:
- More frequent testing of that automation
- No need to run Android Betas separately, by hand

We'd only push to the Play store when asked to, of course.

Comment 7

[[e-mail] (:bhearsum)]

Rough sketch of what needs to be done to get release builds signed on demand:
- signscript.py needs its ifdef block updated: https://github.com/mozilla/build-tools/blob/master/release/signing/signscript.py#L56
- signing.py needs an android_signpackage method, a la the existing ones like dmg_signpackage: https://github.com/mozilla/build-tools/blob/master/release/signing/signing.py#L880
-- Most of the contents of that function will probably be similar to https://github.com/mozilla/mozharness/blob/master/scripts/sign_android.py#L275
- Update https://github.com/mozilla/mozharness/blob/master/scripts/sign_android.py#L275 to talk with the signing server instead of doing things itself
- Update the build system to set MOZ_EXTERNAL_SIGNING_FORMAT for Android builds: https://github.com/mozilla/mozilla-central/blob/master/toolkit/mozapps/installer/signing.mk, maybe some other glue.

Not necessarily as part of this, we should update the nightly android signing to use the signing server instead of on-slave keys. That code is here: https://github.com/mozilla/build-tools/blob/master/release/signing/mozpass.py

There might be other things that need updating, but that's a rough outline.

Comment 8

[Justin Wood (:Callek)]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> Marc volunteered to have a look at this if he has time.

Marc, are you still planning to work on this/are working on this, or should we reassign back to our own pool?

Comment 9

[Chris AtLee [:catlee]]

Attachment: Use MOZ_SIGN_CMD to sign JAR files if set

I think this is all that's needed in the build system.

MAR_SIGN_CMD is something like
python .../signtool.py -t token -n nonce -H signing1.build.mozilla.org:1234

so this expands out to
python .../signtool.py -t token -n nonce -H signing1.build.mozilla.org:1234 -f jar gecko.apk

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 681061 [details] [diff] [review]
Use MOZ_SIGN_CMD to sign JAR files if set

Not an expert here, but this looks right.

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 681061 [details] [diff] [review]
Use MOZ_SIGN_CMD to sign JAR files if set

https://hg.mozilla.org/integration/mozilla-inbound/rev/b178e727b84d

Comment 12

[Chris AtLee [:catlee]]

Still left to do:

Update android-* platform dicts to set 'android_signing': False, remove the android keystore stuff from mock_copyin_files, and set the dep/nightly_signing_servers. Something like this:

diff --git a/mozilla/config.py b/mozilla/config.py
index 5927d95..00548ac 100644
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -648,13 +648,13 @@ PLATFORM_VARS = {
             'update_platform': 'Android_arm-eabi-gcc3',
             'enable_ccache': True,
             'enable_shared_checkouts': True,
+            'nightly_signing_servers': 'dep-signing',
+            'dep_signing_servers': 'dep-signing',
             'use_mock': True,
             'mock_target': 'mozilla-centos6-i386',
             'mock_packages': ['autoconf213', 'mozilla-python27-mercurial', 'ccache', 'android-sdk15', 'android-sdk16', 'android-ndk5', 'zip', 'java-1.6.0-openjdk-devel', 'zlib-devel', 'glibc-static', 'openssh-clients', "mpfr"],
             'mock_copyin_files': [
                 ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
-                ('/home/cltbld/.android', '/builds/.android'),
-                ('/home/cltbld/.mozpass.cfg', '/builds/.mozpass.cfg'),
                 ('/home/cltbld/.hgrc', '/builds/.hgrc'),
             ],
             'env': {
@@ -680,7 +680,7 @@ PLATFORM_VARS = {
             'unittest_masters': GLOBAL_VARS['unittest_masters'],
             'stage_platform': "android",
             'stage_product': 'mobile',
-            'android_signing': True,
+            'android_signing': False,
             'post_upload_include_platform': True,
             'is_mobile_l10n': True,
             'l10n_chunks': 5,
@@ -706,13 +706,13 @@ PLATFORM_VARS = {
             'update_platform': 'Android_arm-eabi-gcc3-armv6',
             'enable_ccache': True,
             'enable_shared_checkouts': True,
+            'nightly_signing_servers': 'dep-signing',
+            'dep_signing_servers': 'dep-signing',
             'use_mock': True,
             'mock_target': 'mozilla-centos6-i386',
             'mock_packages': ['autoconf213', 'mozilla-python27-mercurial', 'ccache', 'android-sdk15', 'android-sdk16', 'android-ndk5', 'zip', 'java-1.6.0-openjdk-devel', 'zlib-devel', 'glibc-static', 'openssh-clients', "mpfr", "bc"],
             'mock_copyin_files': [
                 ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
-                ('/home/cltbld/.android', '/builds/.android'),
-                ('/home/cltbld/.mozpass.cfg', '/builds/.mozpass.cfg'),
                 ('/home/cltbld/.hgrc', '/builds/.hgrc'),
             ],
             'env': {

+BRANCHES['mozilla-central']['platforms']['android']['nightly_signing_servers'] = 'nightly-signing'
+BRANCHES['mozilla-central']['platforms']['android-armv6']['nightly_signing_servers'] = 'nightly-signing'

TBD: Uplift the gecko change (https://hg.mozilla.org/integration/mozilla-inbound/rev/b178e727b84d) to all branches; or modify branch configs in config.py so the changes mentioned above are specific to m-c for now, and ride the trains.

Also TBD: release automation changes

Comment 13

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/b178e727b84d

Comment 14

[Aki Sasaki (not active)]

Attachment: (mozharness) fix android single locale ndk mock packages

Comment 15

[Aki Sasaki (not active)]

Attachment: (custom) enable nightly android l10n signing

This requires buildbot-configs changes, since

File "/src/android-signing/buildbotcustom/steps/signing.py", line 56, in _    _init__
self.servers = list(servers)
TypeError: 'NoneType' object is not iterable

Comment 16

[Aki Sasaki (not active)]

Attachment: (custom) with --signing-server

Comment 17

[Aki Sasaki (not active)]

Attachment: (custom) without --signing-server

Comment 18

[Aki Sasaki (not active)]

Attachment: (mozharness) remove JARSIGNER

Comment 19

[Aki Sasaki (not active)]

Attachment: (mozharness) MOZ_SIGN_CMD for mobile_l10n

Comment 20

[Aki Sasaki (not active)]

Attachment: (custom) with release.py changes

Comment 21

[[e-mail] (:bhearsum)]

With a couple of changes to the mozharness patches, we've been able to sign:
* nightly/dep en-US+multi
* nightly l10n
* release en-US+multi

We still need to make sure that:
* release l10n is signable
* we didn't break b2g_build.py

Comment 22

[Aki Sasaki (not active)]

Attachment: (mozharness) combined mobile_l10n patches

Added --key-alias so we can default to 'nightly' but override to 'release' for releases. (signature verification)

Comment 23

[Aki Sasaki (not active)]

Attachment: (custom) with --key-alias

Comment 24

[Aki Sasaki (not active)]

Attachment: (mozharness) with key_alias in mh configs

This bug may be the perfect argument for me learning how to git rebase patches.

Comment 25

[Aki Sasaki (not active)]

Attachment: (mozharness) tested patch

Comment 26

[[e-mail] (:bhearsum)]

Attachment: add signing servesr for android; enable release signing; drop mozpass stuff

Comment 27

[Chris AtLee [:catlee]]

Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Review of attachment 703442 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-beta.py
@@ +139,5 @@
>          '--summary',
>      ]
>  }
> +releaseConfig['enableSigningAtBuildTime'] = True
> +releaseConfig['enablePartialMarsAtBuildTime'] = True

don't do that!

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Landed, without the partial mars at build time bit.

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 702879 [details] [diff] [review]
(custom) with release.py changes

http://hg.mozilla.org/build/buildbotcustom/rev/017e09be6326

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 701947 [details] [diff] [review]
(mozharness) fix android single locale ndk mock packages

http://hg.mozilla.org/build/mozharness/rev/9555adc60ae6

Comment 31

[[e-mail] (:bhearsum)]

I had to add fake entries into the build scheduler master's passwords.py for the dep/nightly signing servers (it previously only had fake release signing entries).

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Backed out for multiple levels of bustage.

Comment 33

[[e-mail] (:bhearsum)]

Two things that broke:
Wrong python in MOZ_SIGN_CMD for some platforms:
/bin/sh: /tools/python-2.6.5/bin/python: No such file or directory

And for the ones that didn't hit that:
2013-01-18 10:38:37,966 - 94bcc4497f0e63994b16fb5e70c6c4ebaa2fc15e: error uploading file for signing: Unacceptable filename

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

We'll need a new patch with this, plus the original bustage fix, plus a fix for PYTHON26.

Comment 35

[Aki Sasaki (not active)]

Attachment: (configs) fix noion

I can attach an interdiff if needed/wanted.

Comment 36

[[e-mail] (:bhearsum)]

Comment on attachment 702879 [details] [diff] [review]
(custom) with release.py changes

Relanded with the new configs patch. Planning to do a reconfig soon.

Comment 37

[[e-mail] (:bhearsum)]

I've seen all types of Android builds sign successfully on mozilla-inbound. I'm soo happy to have this finally fixed, thanks Aki & Chris!

Comment 38

[[e-mail] (:bhearsum)]

Oh, and I updated the docs to reflect this change:
https://intranet.mozilla.org/RelEngWiki/index.php?title=Signing&action=historysubmit&diff=1522&oldid=1520
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&action=historysubmit&diff=503300&oldid=488456

Comment 39

[Aki Sasaki (not active)]

We should also:

* remove mozpass.cfg and .android from all linux builders
** we can do this easily in puppet, right?
* remove references to mozpass.cfg and .android in new slave / reimage / clean-slave-for-loan docs

New bug? Or finish in this one?

Comment 40

[Aki Sasaki (not active)]

Attachment: (mozharness) remove old sign_android script/configs

hg rm scripts/sign_android.py
hg rm configs/signing

Comment 41

[Rail Aliiev [:rail]]

(In reply to Aki Sasaki [:aki] from comment #39)
> We should also:
> 
> * remove mozpass.cfg and .android from all linux builders
> ** we can do this easily in puppet, right?

Yes. The patch should be reverted once they are gone.

Yup, feel free to file another bug (since this one 
> * remove references to mozpass.cfg and .android in new slave / reimage /

I removed the reference in AWS tools: http://hg.mozilla.org/build/cloud-tools/rev/8996c0ae3467

Comment 42

[Aki Sasaki (not active)]

Comment on attachment 704737 [details] [diff] [review]
(mozharness) remove old sign_android script/configs

http://hg.mozilla.org/build/mozharness/rev/2b64df0af6f6

Comment 43

[Aki Sasaki (not active)]

Comment on attachment 681061 [details] [diff] [review]
Use MOZ_SIGN_CMD to sign JAR files if set

http://hg.mozilla.org/releases/mozilla-release/rev/e1cec6b8a401

Comment 44

[Justin Wood (:Callek)]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/ac6d3c4ab2c2

Comment 45

[Justin Wood (:Callek)]

...and last but not least:
https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/1d3c038a5a73