1642868 - Update verify tests check ja-JP-mac on all platforms now

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Updates, defect, P1)

Description

[Rail Aliiev [:rail]]

After bug 1635103 is landed, the update verify generator https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/release/update-verify-config-creator.py started generating config files with ja-JP-mac included to all platforms.

Examples:

Update verify config generator task: https://firefox-ci-tc.services.mozilla.com/tasks/LdcZR5OvQV6lQD8dO2O4Kw

Failed test: https://firefox-ci-tc.services.mozilla.com/tasks/AlLFFcx0RauBiWJpZxGabg

We have some options:

1. backout bug 1635103. Not ideal. It would be great to stop using shipped-locales in the tests all together.

2. hardcode the ja-JP-mac exception in the config generator

3. stop using shipped-locales and use https://searchfox.org/mozilla-central/source/browser/locales/l10n-changesets.json

4. We can implement 2) and then 3) so we move faster

Comment 1

[Justin Wood (:Callek)]

https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/release/update-verify-config-creator.py#304-313,326-330

Comment 2

[Aki Sasaki (not active)]

https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/l10n/locales.py#97

Comment 3

[Rail Aliiev [:rail]]

A quick fix could be something like https://gist.github.com/rail/4136d2208f6ad9e266c6e8aab22c6678

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Sheriffs were asking about 78.0b3 failures so just to record the full set

• update verify for all platforms, usually chunks 1, 7 or 8, orange on TH
• final verify, blue on TH as it is rerun until it runs out of retries

The root cause is including ja-JP-mac in the list of locales on non-mac, and ja on mac.

Comment 5

[Intermittent Failures Robot]

34 failures in 950 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 34

Platform breakdown:

• linux64-devedition: 2
• linux64-shippable: 2
• linux32-shippable: 2
• linux32-devedition: 2
• windows2012-aarch64-devedition: 2
• windows2012-aarch64-shippable: 2
• osx-shippable: 2
• windows2012-64-devedition: 2
• windows2012-64-shippable: 2
• windows2012-32-shippable: 2
• windows2012-32-devedition: 2
• firefox-release: 6
• osx-cross-devedition: 2
• devedition-release: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1642868&startday=2020-06-04&endday=2020-06-04&tree=all

Comment 6

[Julien Cristau [:jcristau]]

We should get this fixed ASAP.

Comment 7

[Intermittent Failures Robot]

36 failures in 4949 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 36

Platform breakdown:

• firefox-release: 7
• devedition-release: 5
• linux64-devedition: 2
• linux64-shippable: 2
• linux32-shippable: 2
• linux32-devedition: 2
• windows2012-aarch64-devedition: 2
• windows2012-aarch64-shippable: 2
• osx-shippable: 2
• windows2012-64-devedition: 2
• windows2012-64-shippable: 2
• windows2012-32-shippable: 2
• windows2012-32-devedition: 2
• osx-cross-devedition: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1642868&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 8

[Rail Aliiev [:rail]]

Attachment: Bug 1642868 - Update verify tests check ja-JP-mac on all platforms now r=Callek

Comment 9

[Rail Aliiev [:rail]]

Attachment: Bug 1642868 - Update verify tests check ja-JP-mac on all platforms now r=Callek

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/425e0a7a10fd
Update verify tests check ja-JP-mac on all platforms now r=nthomas

Comment 11

[Dorel Luca [:dluca]]

Backed out changeset 425e0a7a10fd (bug 1642868) for Linting failure in /python/mozrelease/mozrelease/platforms.py

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305575842&repo=autoland&lineNumber=510

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=IvSHkus3ROu2ly45pZQ6ww-0&revision=425e0a7a10fdc08fc378a62664b9b8b58eb9bedb

Backout:
https://hg.mozilla.org/integration/autoland/rev/f1c3a24c52bf958b6cd99865ec1aef52b7642452

Comment 12

[Justin Wood (:Callek)]

Attachment: Bug 1642868 - Update verify tests check ja-JP-mac on all platforms now r=jcristau

Beta78 only variant of D78844

Comment 13

[Rail Aliiev [:rail]]

landing the proper fix now...

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/480645112235
Update verify tests check ja-JP-mac on all platforms now r=nthomas

Comment 15

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/9b3d4a819c88

Comment 16

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/480645112235