Closed Bug 1566298 Opened 2 months ago Closed Last month

Sign language packs via Autograph instead of AMO.

Categories

(Release Engineering :: Release Automation: Signing, enhancement)

enhancement
Not set

Tracking

(firefox70 fixed)

RESOLVED FIXED
Tracking Status
firefox70 --- fixed

People

(Reporter: Callek, Assigned: Callek)

References

(Regression)

Details

Attachments

(8 files)

By relying on AMO to do language pack signing we introduced a SPOF onto the release process that is both a long tail as well as a potential issue in terms of finding humans who can help fix the process should AMO be broken for a given l10n changeset.

We treat the in-tree files for L10n as "we expect these to be valid" so we don't need the AMO added validations on them to block releases.

Hey Ritu,

So I have a question that is best on relman, imo.

For this work, should I leave the "AMO" shipping as "let it happen, if it fails sheriffs should notice" or "let it block marking the release as shipped..."

I can see arguments for both. I'm happy to join a zoom meeting with you or the team to discuss further.

Flags: needinfo?(rkothari)

This patch's main goal is to help ease diffs between other parts of this set.

Depends on: 1566197

Justin and I had a good zoom discussion about this.

  1. My understanding is that if the AMO shipping of langpacks fail and there is a major version update (e.g. 68.0b1 -> 69.0b1 or 68.0.1 -> 69.0), beta/release end-users using a Firefox client with a langpack will be reverted to EN-US Firefox client. This would be a terrible experience for an l10n end-user.

  2. I was also told that the likelihood of an AMO upload failure (during releng build promotion workflow) is rare, a handful times a year.

Given 1) and 2), I would like to recommend that we block pushing a release to beta/release channel if the AMO upload fails.

As Justin suggested, if AMO upload fails, it is OK for releng tasks to push build to cdn-test for QA testing but block the balrog sign-off rule creation.

Flags: needinfo?(rkothari)
Attachment #9078307 - Attachment description: Bug 1566298 - Rename release-sign-and-push-langpacks to remove references to sign. r=mtabara Sign language packs via Autograph instead of AMO. → Bug 1566298 - Rename release-sign-and-push-langpacks to remove references to sign. r=mtabara
Attachment #9078308 - Attachment description: Bug 1566298 - Actually sign langpacks via autograph r=mtabara Sign language packs via Autograph instead of AMO. → Bug 1566298 - Actually sign langpacks via autograph r=mtabara
Attachment #9078309 - Attachment description: Bug 1566298 - Temporarily disable langpack beetmover. r=mtabara Sign language packs via Autograph instead of AMO. → Bug 1566298 - Temporarily disable langpack beetmover. r=mtabara
Attachment #9078310 - Attachment description: Bug 1566298 - Add beetmover everywhere for langpacks r=mtabara Sign language packs via Autograph instead of AMO. → Bug 1566298 - Add beetmover everywhere for langpacks r=mtabara

This was due to conversation between myself and :ritu as relman over, documented
at https://bugzilla.mozilla.org/show_bug.cgi?id=1566298#c6

Depends on D37832

Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/497f2b650c1b
Rename release-sign-and-push-langpacks to remove references to sign. r=mtabara
https://hg.mozilla.org/integration/autoland/rev/4b7078499587
Actually sign langpacks via autograph r=mtabara
https://hg.mozilla.org/integration/autoland/rev/7789c2a7decc
Temporarily disable langpack beetmover. r=mtabara
https://hg.mozilla.org/integration/autoland/rev/610ee58dba1a
Add beetmover everywhere for langpacks r=mtabara

I pushed the stack except (to be reviewed) D37832 so I could see this work in production on m-c in the meantime, the remaining patch only affects releases anyway.

Status: RESOLVED → REOPENED
Flags: needinfo?(bugspam.Callek)
Resolution: FIXED → ---

Aki, can you help make sure that the secrets are properly in place re: c#11 - once they are I'll reland the stack.

Flags: needinfo?(aki)

Ah. I added it to the end of the file, after omnija, but failed to remember that there are both release and nightly configs, so that only added it to the release configs. Added to the nightly configs as well, and ran a test run of the above failing l10n task.

Flags: needinfo?(aki)
Depends on: 1573246
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/569f2c9767ec
Rename release-sign-and-push-langpacks to remove references to sign. r=mtabara
https://hg.mozilla.org/integration/autoland/rev/788314e6953a
Actually sign langpacks via autograph r=mtabara
https://hg.mozilla.org/integration/autoland/rev/e777273b4aa5
Temporarily disable langpack beetmover. r=mtabara
https://hg.mozilla.org/integration/autoland/rev/aa829090c7bb
Add beetmover everywhere for langpacks r=mtabara
https://hg.mozilla.org/integration/autoland/rev/3fdd496723be
Block explicit scheduling for the release in balrog on AMO langpack pushing. r=mtabara
Flags: needinfo?(bugspam.Callek)
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b1eacaaab258
Fix release regression from Bug 1566298 - use 'en-US' so that tasks know to strip it out from path when looking for it in upstream tasks. r=mtabara

(In reply to Razvan Maries from comment #22)

Aki, langpack is failing on the fix https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&searchStr=langpack&selectedJob=263635813 with another error: https://tools.taskcluster.net/groups/M_5Ka6RQSN2DDiMSmVTe8w/tasks/arqpOHXYTMem5eRU13wlNg/runs/0/logs/public%2Flogs%2Fchain_of_trust.log

seems like we knew about moving target.langpack.xpi out of en-US based on: https://hg.mozilla.org/releases/mozilla-beta/rev/b1eacaaab2581c4b2f2068b5543a9f988ffd69a8

however, we are still hitting issues where beetmover expects target.langpack.xpi upstream_artifacts to live in 'en-US'. Maybe this block pointed out by nick was missed? https://hg.mozilla.org/releases/mozilla-beta/file/default/taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py#l144

Flags: needinfo?(aki) → needinfo?(bugspam.Callek)
No longer blocks: 1576793
Regressed by: 1576793
Flags: needinfo?(bugspam.Callek)
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fa71c787eec9
fix langpack-copy dependencies. r=nthomas a=release CLOSED TREE
https://hg.mozilla.org/integration/autoland/rev/9df802613d63
Followup, don't look for en-US langpacks in an 'en-US/' path. r=jlorenzo
Regressions: 1578289
You need to log in before you can comment on or make changes to this bug.