1663130 - android-signing jobs are busted on ESR68

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Signing, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Sherriffs raised this last night. The error message smells like Fennec cleanup leftovers somewhere in ESR68:

    a = get_autograph_config(context.autograph_configs, cert_type, [fmt], raise_on_empty=True)
  File "/app/lib/python3.8/site-packages/signingscript/sign.py", line 143, in get_autograph_config
    raise SigningScriptError(f"No autograph config found with cert type {cert_type} and formats {signing_formats}")
signingscript.exceptions.SigningScriptError: No autograph config found with cert type project:releng:signing:cert:release-signing and formats ['autograph_apk_fennec_sha1']

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

I checked the overall graph where we're seeing this and it's the on-push CI graphs for ESR68, e.g. last one is here.

I think this is a fallout from bug 1608874, where the overall Fennec cleanup has been completed.

Looking at the failed signing jobs, they are present only in the last commit on ESR68, which dates from Thu, Sep 3, 16:47:41. The last one from Thu, Aug 20, 21:19:18 was green so something changed in between. I dug into the signingscript changes and turns out we recently landed a couple of cleanup PRs, such as this, this and this.

Out of them, the second one removed the ability for Fennec nightly/release to sign with autograph_apk_fennec_sha1, which is causing the error we're seeing in comment 0.

Even though the PR landed on the 3rd of August, turns out it wasn't deployed until three days ago. Looking into Dockerhub logs for all the production- images that we've pushed, the latest one is production-20200901144002-7ad4dca957626fb66ddbedc79dfa058440cddba5 which was pushed (as the tag says) on the 1st of September. The second one in the logs is production-12.2.0-20200622150052-91496c162605a73229043563bc9d66e6821a98c0 which was pushed on the 22nd of June. So this explains how come we haven't seen the ESR68 android-nightly bustages earlier.

Bug 1608874's https://hg.mozilla.org/releases/mozilla-esr68/rev/0176d50e188d removed the Fennec promotion graphs from ESR68 altogether, but the build + signing jobs are part of the on-push graphs, not the promotion phase, so likely they're still lingering somewhere in-tree.

My suspicion is:
a) we can either ignore the android-builds until we EOL ESR68, since we're now fully migrated to Fenix so we won't build another Fennec
b) we can remove the android build + signing logic altogether on esr68 to prevent these from running and burning resources

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Since ESR68 will be EOL in less than three weeks and most of the commits in ESR68 are DONTBUILD, I'm tempted to believe we'll opt for a) from comment 1 above. However, if we opt for b), I think we'll have to remove android build + signing altogether, or change the signing to a different format in here

Leaving a NI for Johan.

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1663130 - Remove fennec-release builds r=RyanVM,mtabara

Comment 4

[Johan Lorenzo [:jlorenzo]]

I agree and thank you Mihai for documenting all this!

To be more precise, this was removed with [1]. I don't think we should sign anything with the release key anymore, even for CI. I'm okay to keep dep-signing around.

So, I believe it's safe to go with option b). I just put a patch which works locally. I'm okay to go with option a) if we don't think my patch is safe enough 🙂

[1] https://github.com/mozilla-releng/scriptworker-scripts/pull/247/files#diff-14f1a7b81d2f0f9986a3fbe12661dff2L196

Comment 5

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9174210 [details]
Bug 1663130 - Remove fennec-release builds r=RyanVM,mtabara

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: ESR-only patch
• User impact if declined: Sheriffs will keep seeing red signing jobs
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It just removes the fennec release flavor which should not be used anymore. This patch was r+'d by :RyanVM.
• String or UUID changes made by this patch: None

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9174210 [details]
Bug 1663130 - Remove fennec-release builds r=RyanVM,mtabara

Sure, go for it.

Comment 7

[Julien Cristau [:jcristau] (back April 22)]

https://hg.mozilla.org/releases/mozilla-esr68/rev/f72861bb0a1c7914e5f6af56e632f2197c1cd984