1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

Maple[1] uses dep-signing which makes pushapk_scriptworker failing. We need a staging instance of pushapk that can't push to prod, but supports dep-signing.

[1] https://hg.mozilla.org/projects/maple/

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: [pushapkscript] Support artifacts signed by dep-signing

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review commit: https://reviewboard.mozilla.org/r/201250/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/201250/

Comment 3

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201250/diff/1-2/

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozapkpublisher] push_apk.py: expose --do-not-contact-google-play

mozapkpublisher will error out at the first error reported by Google Play. I introduced a mode that keeps every step but does not do anything with Google Play.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201250/diff/2-3/

Comment 6

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201250/diff/3-4/

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: [pushapkscript] Enforce Google Play not being contacted if not a on known channel

Comment 8

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201250/diff/4-5/

Comment 9

[Johan Lorenzo [:jlorenzo]]

Attachment: [aws-manager] Script used to spawn dep-pushapkworker EC2 instance

Status update

It's been about year since I deployed pushapkworker-1 (bug 1307826). When I read the bug up, I realized a few things were missing.

1. There is no need to connect to inventory[1] to reserve a domain name (FQDN). For some reason, I thought I did something similar last year. Anyway, it wasn't needed last week. I just ran the script (attached), which handled the DNS subscription just fine. This script is a modified copy of the one Aki used to spawn dep-signing scriptworkers. I noticed he used the same configuration as the prod workers[2]. This means machines live under the same subnet and samesec group as production ones. Then, I did the same for dep-pushapkworker


2. You don't need to create new Chain of Trust GPG keys, if you set verify_chain_of_trust to true, but not the 2 other vars (namely sign_chain_of_trust and verify_cot_signature). This still lets the scriptworker instance to download upstream artifacts.


3. If the FQDN doesn't match any node in the production listing, you still may connect to the machine and attach it to a personal puppet environment. In this regard, I did:
> ssh [e-mail] -i /PATH/TO/PRIVATE/REPO/keys/aws-releng
> puppet agent --test --server releng-puppet2.srv.releng.scl3.mozilla.com --environment jlorenzo
Once this puppet run finished, I was able to connect with my regular handle. 



Then I discovered:

A. Puppet is still running v3.7. This prevents the use of loops, unless you globally activate the "future parser"[4]. Then, I rewrote my puppet patch with switch statements to define the right variables. 


B. No need to store the APKs certificates on Hiera. They are definitely public, and they can be extracted from the APKs. For instance, I got the dep certificate via:
> unzip -p target.apk META-INF/DEP.RSA | openssl pkcs7 -inform DER -print_certs -out nightly.pem
I confirmed the certs are right ones, by diff'ing nightly and release against the ones already imported in the prod push-apk. I versioned the certificate in the puppet repo, directly. 


C. I will delete the "dev" TC client[5] in favor of a "dep" one. The rationale is:
 * we shouldn't have a dev instance that touches the production APKs.
 * we shouldn't have any non-production APK signed with the production keys. 
 * non-production APKs are signed with "dep"
 * pushapkscript relies heavily on what key signed the APK.
 * ergo non-production pushapk_scriptworker must be called "dep" and there should no "dev" instance (for clarity).
Code in puppet has been changed accordingly.


D. Some parts of pushapkscript and mozapkpublisher print out they uploaded something on Google Play. In order to reduce the confusion, I made sure these log lines say they processed dummy things. I also added some forewords at the top of the execution of pushapkscript to recall what will happen whether we're on dep/prod or whether the commit option (which used to be called "dry-run") is turned on.



[1] https://inventory.mozilla.org/en-US/core/search/#q=pushapkworker
[2] https://github.com/mozilla-releng/build-cloud-tools/blob/bb5dda8b19b400e4ced61b49d0c33bcba21c6311/configs/depsigning-worker and https://github.com/mozilla-releng/build-cloud-tools/blob/bb5dda8b19b400e4ced61b49d0c33bcba21c6311/configs/signingworker. FIles are different but content is the same. 
[3] https://dxr.mozilla.org/build-central/rev/ca31b2a85d8fa313a245acde72021b8dc8744837/relabs-puppet/modules/signing_scriptworker/manifests/settings.pp#30
[4] https://docs.puppet.com/puppet/3.7/function.html#each and https://docs.puppet.com/puppet/3.7/experiments_future.html#enabling-the-future-parser
[5] https://tools.taskcluster.net/auth/clients/project%2Freleng%2Fscriptworker%2Fpushapk%2Fdev

Comment 10

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930881 [details] [review]
[mozapkpublisher] push_apk.py: expose --do-not-contact-google-play

Patch tested in integration with the rest of the stack in: https://tools.taskcluster.net/groups/dpTLsHJ2Ts6I-yy_4jKV1w/tasks/OIAbcNBRSyOEO1H0n_ftMQ/runs/8/logs/public%2Flogs%2Flive_backing.log

Comment 11

[Johan Lorenzo [:jlorenzo]]

Bug 1420459 being done, I let dep-pushapk handle the task, with the right worker group and TC client[1]. Then, I deleted the old (and confusing) dev TC client[2].

What's left:
1. Land the mozapkpublisher/pushapkscript patches.
2. Release mozapkpublisher 0.5.0 and pushapkscript 0.4.0
3. Update the puppet patch to use these packages.
4. Get a review on the puppet patch.
5. Clean the staging mozapkpublisher==0.5.0-13a7be71ace8 and pushapkscript==0.4.0-2a9381d332a5 python packages from releng-puppet2
6. Land puppet in production and stop dep-pushapkworker-1 from being attached to my personal account. 


[1] https://tools.taskcluster.net/groups/H2ukXbn4Q1KMReCxVaYlMg/tasks/ISmfIBIkTfShf8eNYm0BKg/runs/0
[2] https://tools.taskcluster.net/auth/clients/project%2Freleng%2Fscriptworker%2Fpushapk%2Fdev - now a 404

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1412836 - Use dep-pushapk scritpworker for non m-c, m-b, m-r branches

Review commit: https://reviewboard.mozilla.org/r/203486/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/203486/

Comment 13

[Johan Lorenzo [:jlorenzo]]

I forgot:

(In reply to Johan Lorenzo [:jlorenzo] from comment #11) 
> What's left:
0. Backport changes on the maple branch to m-c
...
7. Land the in-tree changes to autoland and let it reach m-c.
8. Profit!

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8932053 [details] [review]
[pushapkscript] Enforce Google Play not being contacted if not a on known channel

Addressed comments in the PR.

Comment 15

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930881 [details] [review]
[mozapkpublisher] push_apk.py: expose --do-not-contact-google-play

:mtabara r+'d at [1]. Landed on master at [2]

[1] https://github.com/mozilla-releng/mozapkpublisher/pull/48#pullrequestreview-79473400
[2] https://github.com/mozilla-releng/mozapkpublisher/commit/9b6e1e11f49afe154d446b9303006197946d8617

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8932053 [details] [review]
[pushapkscript] Enforce Google Play not being contacted if not a on known channel

Addressed tiny comments in the PR.

Comment 17

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201250/diff/5-6/

Comment 18

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8932053 [details] [review]
[pushapkscript] Enforce Google Play not being contacted if not a on known channel

https://github.com/mozilla-releng/pushapkscript/commit/7d5463f6d90c5a69993ca647610d63124da1834b

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

https://reviewboard.mozilla.org/r/201250/#review209080

::: modules/pushapk_scriptworker/manifests/init.pp:88
(Diff revision 6)
>              cot_job_type             => 'pushapk',
>  
> +            sign_chain_of_trust      => $pushapk_scriptworker::settings::sign_chain_of_trust,
> +            verify_chain_of_trust    => $pushapk_scriptworker::settings::verify_chain_of_trust,
> +            verify_cot_signature     => $pushapk_scriptworker::settings::verify_cot_signature,
> +

Interesting. I went with `env_config` since that seems more straightforward to me. If the switch statement works for you, that's cool.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 8932437 [details]
Bug 1412836 - Use dep-pushapk scritpworker for non m-c, m-b, m-r branches

https://reviewboard.mozilla.org/r/203486/#review209092

This looks good. However, because we're switching from `dry-run` to `commit`, we're likely going to need to support both task definition formats until this patch rides the trains into m-r.

Comment 21

[Johan Lorenzo [:jlorenzo]]

Attachment: [pushapkscript] Deprecate dry_run flag in task payload, instead of killing it

Great point, Aki. Thank you for spotting it. I made a follow up patch that still allows dry-run.

Comment 22

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8932793 [details] [review]
[pushapkscript] Deprecate dry_run flag in task payload, instead of killing it

Mihai reviewed this patch at https://github.com/mozilla-releng/pushapkscript/pull/31#pullrequestreview-79791380

Landed at: https://github.com/mozilla-releng/pushapkscript/commit/72e3c67cdf3f56cfef04204257e9e16bcb0b8634

Comment 23

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8930072 [details]
[build/puppet] Bug 1412836 - Setup a pushapk_scriptworker instance that supports dep-signing

Landed on:
* default https://hg.mozilla.org/build/puppet/rev/f785fcec174f9039797b276415dcde483d7002f6
* production: https://hg.mozilla.org/build/puppet/rev/a5ca7abd0121abf75e65832776deca2f28672feb

Comment 24

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8923375 [details] [review]
[pushapkscript] Support artifacts signed by dep-signing

Fixed silly linter issues on:
* default: https://hg.mozilla.org/build/puppet/rev/bc00cd01968297289e882f4108d3962734f713ac
* production: https://hg.mozilla.org/build/puppet/rev/a5eef6de0ad90396bef4e00ab4030c7628fd39f1

Comment 25

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #11)
Steps 0 to 6 done. All the patches landed worked on maple against the prod config of dep-pushapk. https://tools.taskcluster.net/groups/H2ukXbn4Q1KMReCxVaYlMg/tasks/ISmfIBIkTfShf8eNYm0BKg/runs/3

Time to get the in-tree patch landed.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c5b5263f3c1c
Use dep-pushapk scritpworker for non m-c, m-b, m-r branches r=aki

Comment 27

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c5b5263f3c1c

Comment 28

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8932437 [details]
Bug 1412836 - Use dep-pushapk scritpworker for non m-c, m-b, m-r branches

The puppet patch didn't break the production instance of push-apk: https://tools.taskcluster.net/groups/LfAtlzm5Ro6nq0DNy4TbbA/tasks/DYTFuSrPQvmh39CGRB_cJw/runs/0/logs/public%2Flogs%2Flive_backing.log. See line:
> 2017-11-30 00:59:46,903 - pushapkscript.script - WARNING - You will publish APKs to Google Play. This action is irreversible,if no error is detected either by this script or by Google Play.

That line was added in attachment 8932053 [details] [review].

Then, this bug is officially resolved.