Closed Bug 705807 Opened 8 years ago Closed 7 years ago

Android signing-on-demand

Categories

(Release Engineering :: General, defect, P3)

ARM
Android
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: catlee)

References

Details

(Whiteboard: [android][signing][automation][leave open])

Attachments

(6 files, 9 obsolete files)

638 bytes, patch
aki
: review+
ted
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
8.63 KB, patch
catlee
: review+
Details | Diff | Splinter Review
3.12 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
24.96 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
21.64 KB, patch
bhearsum
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
28.73 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
https://bugzilla.mozilla.org/show_bug.cgi?id=705214#c1

As noted in that bug, not producing nightly-signed bits will break tegra testing.
I *think* we can get around this, but it might not be possible due to Android permissions.  We need to verify that only producing release-signed bits will work from a testing perspective, or generate both nightly- and release-signed bits before we can turn this on.
Depends on: 706243
Multilocale will not be going away (Amazon Market), so most likely our requirements for an Android release build will be

a) nightly-signed en-US apk
b) release-signed en-US apk
c) nightly-signed multi apk
d) release-signed multi apk
We may as well expand this for nightlies, so we have a defined and similar way to sign for both nightlies and releases.
Summary: Android signing-on-demand (for releases) → Android signing-on-demand
Whiteboard: [android][signing][releases] → [android][signing][automation]
(In reply to Aki Sasaki [:aki] from comment #0)
> https://bugzilla.mozilla.org/show_bug.cgi?id=705214#c1
> 
> As noted in that bug, not producing nightly-signed bits will break tegra
> testing.

I'm going to revisit a lot of this while porting remote talos/unittests to mozharness.
I think I'm going to re-sign every apk with a test-specific key before installing, which will allow us to create only nightly signed bits during nightly jobs and only release signed bits during release jobs.

(In reply to Aki Sasaki [:aki] from comment #1)
> Multilocale will not be going away (Amazon Market)

Multilocale is now only enabled on android-xul.
When we shift entirely to android native, there will be no more multilocale.

I think we're unblocked here from these two perspectives.
Component: Release Engineering → Release Engineering: Automation
Priority: P5 → P3
QA Contact: release → catlee
Marc volunteered to have a look at this if he has time.
Assignee: nobody → marc.jessome
Since daily mobile betas are not a part of the rapid betas plan, I don't believe this is a blocker as noted on [1].

[1] https://wiki.mozilla.org/Release_Management/Rapid_Betas#RelEng.2FIT
We've been talking about the possibility of building Android as part of rapid/automated betas for a couple of reasons:
- More frequent testing of that automation
- No need to run Android Betas separately, by hand

We'd only push to the Play store when asked to, of course.
Rough sketch of what needs to be done to get release builds signed on demand:
- signscript.py needs its ifdef block updated: https://github.com/mozilla/build-tools/blob/master/release/signing/signscript.py#L56
- signing.py needs an android_signpackage method, a la the existing ones like dmg_signpackage: https://github.com/mozilla/build-tools/blob/master/release/signing/signing.py#L880
-- Most of the contents of that function will probably be similar to https://github.com/mozilla/mozharness/blob/master/scripts/sign_android.py#L275
- Update https://github.com/mozilla/mozharness/blob/master/scripts/sign_android.py#L275 to talk with the signing server instead of doing things itself
- Update the build system to set MOZ_EXTERNAL_SIGNING_FORMAT for Android builds: https://github.com/mozilla/mozilla-central/blob/master/toolkit/mozapps/installer/signing.mk, maybe some other glue.

Not necessarily as part of this, we should update the nightly android signing to use the signing server instead of on-slave keys. That code is here: https://github.com/mozilla/build-tools/blob/master/release/signing/mozpass.py

There might be other things that need updating, but that's a rough outline.
(In reply to Ben Hearsum [:bhearsum] from comment #4)
> Marc volunteered to have a look at this if he has time.

Marc, are you still planning to work on this/are working on this, or should we reassign back to our own pool?
I think this is all that's needed in the build system.

MAR_SIGN_CMD is something like
python .../signtool.py -t token -n nonce -H signing1.build.mozilla.org:1234

so this expands out to
python .../signtool.py -t token -n nonce -H signing1.build.mozilla.org:1234 -f jar gecko.apk
Attachment #681061 - Flags: review?(ted)
Attachment #681061 - Flags: review?(aki)
Attachment #681061 - Flags: feedback?
Comment on attachment 681061 [details] [diff] [review]
Use MOZ_SIGN_CMD to sign JAR files if set

Not an expert here, but this looks right.
Attachment #681061 - Flags: review?(aki) → review+
Attachment #681061 - Flags: feedback?
Attachment #681061 - Flags: review?(ted) → review+
Comment on attachment 681061 [details] [diff] [review]
Use MOZ_SIGN_CMD to sign JAR files if set

https://hg.mozilla.org/integration/mozilla-inbound/rev/b178e727b84d
Attachment #681061 - Flags: checked-in+
Whiteboard: [android][signing][automation] → [android][signing][automation][leave open]
Assignee: marc.jessome → catlee
Still left to do:

Update android-* platform dicts to set 'android_signing': False, remove the android keystore stuff from mock_copyin_files, and set the dep/nightly_signing_servers. Something like this:

diff --git a/mozilla/config.py b/mozilla/config.py
index 5927d95..00548ac 100644
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -648,13 +648,13 @@ PLATFORM_VARS = {
             'update_platform': 'Android_arm-eabi-gcc3',
             'enable_ccache': True,
             'enable_shared_checkouts': True,
+            'nightly_signing_servers': 'dep-signing',
+            'dep_signing_servers': 'dep-signing',
             'use_mock': True,
             'mock_target': 'mozilla-centos6-i386',
             'mock_packages': ['autoconf213', 'mozilla-python27-mercurial', 'ccache', 'android-sdk15', 'android-sdk16', 'android-ndk5', 'zip', 'java-1.6.0-openjdk-devel', 'zlib-devel', 'glibc-static', 'openssh-clients', "mpfr"],
             'mock_copyin_files': [
                 ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
-                ('/home/cltbld/.android', '/builds/.android'),
-                ('/home/cltbld/.mozpass.cfg', '/builds/.mozpass.cfg'),
                 ('/home/cltbld/.hgrc', '/builds/.hgrc'),
             ],
             'env': {
@@ -680,7 +680,7 @@ PLATFORM_VARS = {
             'unittest_masters': GLOBAL_VARS['unittest_masters'],
             'stage_platform': "android",
             'stage_product': 'mobile',
-            'android_signing': True,
+            'android_signing': False,
             'post_upload_include_platform': True,
             'is_mobile_l10n': True,
             'l10n_chunks': 5,
@@ -706,13 +706,13 @@ PLATFORM_VARS = {
             'update_platform': 'Android_arm-eabi-gcc3-armv6',
             'enable_ccache': True,
             'enable_shared_checkouts': True,
+            'nightly_signing_servers': 'dep-signing',
+            'dep_signing_servers': 'dep-signing',
             'use_mock': True,
             'mock_target': 'mozilla-centos6-i386',
             'mock_packages': ['autoconf213', 'mozilla-python27-mercurial', 'ccache', 'android-sdk15', 'android-sdk16', 'android-ndk5', 'zip', 'java-1.6.0-openjdk-devel', 'zlib-devel', 'glibc-static', 'openssh-clients', "mpfr", "bc"],
             'mock_copyin_files': [
                 ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
-                ('/home/cltbld/.android', '/builds/.android'),
-                ('/home/cltbld/.mozpass.cfg', '/builds/.mozpass.cfg'),
                 ('/home/cltbld/.hgrc', '/builds/.hgrc'),
             ],
             'env': {

+BRANCHES['mozilla-central']['platforms']['android']['nightly_signing_servers'] = 'nightly-signing'
+BRANCHES['mozilla-central']['platforms']['android-armv6']['nightly_signing_servers'] = 'nightly-signing'

TBD: Uplift the gecko change (https://hg.mozilla.org/integration/mozilla-inbound/rev/b178e727b84d) to all branches; or modify branch configs in config.py so the changes mentioned above are specific to m-c for now, and ride the trains.

Also TBD: release automation changes
Attachment #701947 - Flags: review?(catlee) → review+
This requires buildbot-configs changes, since

File "/src/android-signing/buildbotcustom/steps/signing.py", line 56, in _    _init__
self.servers = list(servers)
TypeError: 'NoneType' object is not iterable
Attached patch (custom) with --signing-server (obsolete) — Splinter Review
Attachment #701955 - Attachment is obsolete: true
Attached patch (mozharness) remove JARSIGNER (obsolete) — Splinter Review
Attachment #702838 - Attachment is obsolete: true
Attachment #702848 - Attachment is obsolete: true
With a couple of changes to the mozharness patches, we've been able to sign:
* nightly/dep en-US+multi
* nightly l10n
* release en-US+multi

We still need to make sure that:
* release l10n is signable
* we didn't break b2g_build.py
Added --key-alias so we can default to 'nightly' but override to 'release' for releases. (signature verification)
Attachment #702850 - Attachment is obsolete: true
Attachment #702868 - Attachment is obsolete: true
Attached patch (custom) with --key-alias (obsolete) — Splinter Review
Attachment #702879 - Attachment is obsolete: true
Attachment #702967 - Attachment is obsolete: true
Attachment #702879 - Attachment is obsolete: false
This bug may be the perfect argument for me learning how to git rebase patches.
Attachment #702965 - Attachment is obsolete: true
Attachment #702972 - Attachment is obsolete: true
Attachment #703323 - Flags: review?(catlee)
Attachment #703323 - Flags: review?(catlee) → review+
Attachment #702879 - Flags: review?(catlee)
Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Review of attachment 703442 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-beta.py
@@ +139,5 @@
>          '--summary',
>      ]
>  }
> +releaseConfig['enableSigningAtBuildTime'] = True
> +releaseConfig['enablePartialMarsAtBuildTime'] = True

don't do that!
Attachment #703442 - Flags: review?(catlee) → review+
Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Landed, without the partial mars at build time bit.
Attachment #703442 - Flags: checked-in+
Attachment #702879 - Flags: review?(catlee) → review+
Comment on attachment 701947 [details] [diff] [review]
(mozharness) fix android single locale ndk mock packages

http://hg.mozilla.org/build/mozharness/rev/9555adc60ae6
Attachment #701947 - Flags: checked-in+
Attachment #703323 - Flags: checked-in+
I had to add fake entries into the build scheduler master's passwords.py for the dep/nightly signing servers (it previously only had fake release signing entries).
Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

Backed out for multiple levels of bustage.
Attachment #703442 - Flags: checked-in+ → checked-in-
Attachment #703323 - Flags: checked-in+ → checked-in-
Attachment #702879 - Flags: checked-in+ → checked-in-
Two things that broke:
Wrong python in MOZ_SIGN_CMD for some platforms:
/bin/sh: /tools/python-2.6.5/bin/python: No such file or directory

And for the ones that didn't hit that:
2013-01-18 10:38:37,966 - 94bcc4497f0e63994b16fb5e70c6c4ebaa2fc15e: error uploading file for signing: Unacceptable filename
Comment on attachment 703442 [details] [diff] [review]
add signing servesr for android; enable release signing; drop mozpass stuff

We'll need a new patch with this, plus the original bustage fix, plus a fix for PYTHON26.
Attachment #703442 - Attachment is obsolete: true
I can attach an interdiff if needed/wanted.
Attachment #703938 - Flags: review?(bhearsum)
Attachment #703938 - Flags: review?(bhearsum) → review+
Comment on attachment 702879 [details] [diff] [review]
(custom) with release.py changes

Relanded with the new configs patch. Planning to do a reconfig soon.
Attachment #702879 - Flags: checked-in- → checked-in+
Attachment #703938 - Flags: checked-in+
Attachment #703323 - Flags: checked-in- → checked-in+
I've seen all types of Android builds sign successfully on mozilla-inbound. I'm soo happy to have this finally fixed, thanks Aki & Chris!
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
We should also:

* remove mozpass.cfg and .android from all linux builders
** we can do this easily in puppet, right?
* remove references to mozpass.cfg and .android in new slave / reimage / clean-slave-for-loan docs

New bug? Or finish in this one?
hg rm scripts/sign_android.py
hg rm configs/signing
Attachment #704737 - Flags: review?(bhearsum)
Attachment #704737 - Flags: review?(bhearsum) → review+
(In reply to Aki Sasaki [:aki] from comment #39)
> We should also:
> 
> * remove mozpass.cfg and .android from all linux builders
> ** we can do this easily in puppet, right?

Yes. The patch should be reverted once they are gone.

Yup, feel free to file another bug (since this one 
> * remove references to mozpass.cfg and .android in new slave / reimage /

I removed the reference in AWS tools: http://hg.mozilla.org/build/cloud-tools/rev/8996c0ae3467
Comment on attachment 704737 [details] [diff] [review]
(mozharness) remove old sign_android script/configs

http://hg.mozilla.org/build/mozharness/rev/2b64df0af6f6
Attachment #704737 - Flags: checked-in+
Blocks: 833524
Blocks: 833861
Depends on: 833997
Blocks: 835073
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.