1746641 - Perma macOS Late Beta L10N bustage IndexError: list index out of range | make[2]: *** [/builds/worker/checkouts/gecko/toolkit/locales/l10n.mk:115: repackage-zip] Error 1 when Gecko 97 switches to Late Beta on 2022-01-21

Status: RESOLVED INVALID

(Firefox Build System :: General, defect, P2)

Description

[Iulian Moraru]

[Tracking Requested - why for this release]:

Central-as-late-beta simulation:
How to run these simulations:

Failure log:

[task 2021-12-17T15:47:13.015Z] 15:47:13     INFO -  make[2]: Entering directory '/builds/worker/workspace/obj-build/browser/locales'
[task 2021-12-17T15:47:13.016Z] 15:47:13     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/toolkit/mozapps/installer/l10n-repack.py '/builds/worker/workspace/obj-build/dist/l10n-stage/firefox/Firefox Nightly.app/Contents/Resources' ../../dist/xpi-stage/locale-ar \
[task 2021-12-17T15:47:13.016Z] 15:47:13     INFO -  	 \
[task 2021-12-17T15:47:13.129Z] 15:47:13     INFO -  Error: Missing file(s): *
[task 2021-12-17T15:47:13.132Z] 15:47:13    ERROR -  Traceback (most recent call last):
[task 2021-12-17T15:47:13.132Z] 15:47:13     INFO -    File "/builds/worker/checkouts/gecko/toolkit/mozapps/installer/l10n-repack.py", line 73, in <module>
[task 2021-12-17T15:47:13.132Z] 15:47:13     INFO -      main()
[task 2021-12-17T15:47:13.132Z] 15:47:13     INFO -    File "/builds/worker/checkouts/gecko/toolkit/mozapps/installer/l10n-repack.py", line 63, in main
[task 2021-12-17T15:47:13.132Z] 15:47:13     INFO -      l10n.repack(
[task 2021-12-17T15:47:13.132Z] 15:47:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozpack/packager/l10n.py", line 306, in repack
[task 2021-12-17T15:47:13.133Z] 15:47:13     INFO -      _repack(app_finder, l10n_finder, copier, formatter, non_chrome)
[task 2021-12-17T15:47:13.133Z] 15:47:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozpack/packager/l10n.py", line 142, in _repack
[task 2021-12-17T15:47:13.133Z] 15:47:13     INFO -      locale = app.locales[0]
[task 2021-12-17T15:47:13.133Z] 15:47:13     INFO -  IndexError: list index out of range
[task 2021-12-17T15:47:13.141Z] 15:47:13    ERROR -  make[2]: *** [/builds/worker/checkouts/gecko/toolkit/locales/l10n.mk:115: repackage-zip] Error 1
[task 2021-12-17T15:47:13.141Z] 15:47:13     INFO -  make[2]: Leaving directory '/builds/worker/workspace/obj-build/browser/locales'
[task 2021-12-17T15:47:13.141Z] 15:47:13    ERROR -  make[1]: *** [/builds/worker/checkouts/gecko/toolkit/locales/l10n.mk:151: repackage-zip-ar] Error 2
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  make[1]: Leaving directory '/builds/worker/workspace/obj-build/browser/locales'
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  make: *** [Makefile:103: installers-ar] Error 2
[task 2021-12-17T15:47:13.142Z] 15:47:13    ERROR - Return code: 2
[task 2021-12-17T15:47:13.142Z] 15:47:13    ERROR - make installers-ar failed
[task 2021-12-17T15:47:13.142Z] 15:47:13    ERROR - failure: repack_locale(ar)
[task 2021-12-17T15:47:13.142Z] 15:47:13    ERROR - # TBPL FAILURE #
[task 2021-12-17T15:47:13.142Z] 15:47:13  WARNING - setting return code to 2
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO - Copying /builds/worker/checkouts/gecko/browser/config/mozconfigs/macosx64/l10n-mozconfig to /builds/worker/checkouts/gecko/.mozconfig
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO - Reading from file /builds/worker/checkouts/gecko/.mozconfig
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO - Contents:
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  . "$topsrcdir/browser/config/mozconfigs/common"
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  . "$topsrcdir/build/mozconfig.no-compile"
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.142Z] 15:47:13     INFO -  if test `uname -s` = "Linux"; then
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -    # We need to indicate the target for cross builds
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -    ac_add_options --target=x86_64-apple-darwin
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -    export MKFSHFS=$MOZ_FETCHES_DIR/hfsplus-tools/newfs_hfs
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -    export DMG_TOOL=$MOZ_FETCHES_DIR/dmg/dmg
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -    export HFS_TOOL=$MOZ_FETCHES_DIR/dmg/hfsplus
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  fi
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  ac_add_options --with-branding=browser/branding/nightly
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  ac_add_options --disable-nodejs
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  unset NODEJS
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  export MOZILLA_OFFICIAL=1
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  # Don't autoclobber l10n, as this can lead to missing binaries and broken builds
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  # Bug 1283438
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  mk_add_options AUTOCLOBBER=
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO -  . "$topsrcdir/build/mozconfig.common.override"
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO - Running command: ['make', 'installers-en-CA'] in /builds/worker/workspace/obj-build/browser/locales
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO - Copy/paste: make installers-en-CA
[task 2021-12-17T15:47:13.143Z] 15:47:13     INFO - Using env: (same as previous command)
[task 2021-12-17T15:47:13.157Z] 15:47:13     INFO -  make[1]: Entering directory '/builds/worker/workspace/obj-build/browser/locales'

Comment 1

[Iulian Moraru]

Hi Hubert! Can you please take a look at this, could this be caused by your latest work in Bug 1745940?
Thank you!

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Unrelated to the bug in comment 1745940.

Comment 3

[Mike Hommey [:glandium]]

What was the revision of the last non-busted central-as-late-beta simulation?

Comment 4

[Mike Hommey [:glandium]]

That simulation doesn't seem to have applied any replacements to mozconfigs, and they're still using nightly branding. Missing a central-to-beta on the command line?

Comment 5

[Andreea Pavel [:apavel]]

(In reply to Mike Hommey [:glandium] from comment #3)

What was the revision of the last non-busted central-as-late-beta simulation?

Last non-busted revision: https://hg.mozilla.org/mozilla-central/rev/792c9fbceb7170c1c00d8bed57a2c3d2cf6d1913
Beta sim before bustage: https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=24d2a6fc5c9fbd1b5cc2968ed58dac8ea868b559

Doesn't show up on latest beta sim, as these jobs didn't run: https://treeherder.mozilla.org/jobs?repo=try&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=56e2fa7ba27e9017a66536fa1cb065bc1efbf0b6&searchStr=l10n&group_state=expanded

Base revision https://hg.mozilla.org/mozilla-central/rev/e6f994394be4fd1eb00bd8db40e8a8dcb42f5f9a

Comment 6

[Intermittent Failures Robot]

1 failures in 3263 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1746641&startday=2021-12-13&endday=2021-12-19&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Mike is correct: The central-as-late-beta simulation for which this got filed only ran the command to switch an early beta config to a late beta one without applying the central-to-early-beta migration first.

But the issue persists with proper late betas.

Comment 8

[Mike Hommey [:glandium]]

This time it's the tasks before the l10n task that still use a nightly branding.

Comment 9

[Mike Hommey [:glandium]]

That push is hitting the problem that it's currently not possible to do a fuzzy try with a release_type set. This is related to bug 1608037.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p2 as the bug is tracked by a release manager for the current nightly.
See What Do You Triage for more information

Comment 11

[Christian Holler (:decoder)]

Will the patch in bug 1608037 solve this and if so, what's the status of that patch?

Comment 12

[Mike Hommey [:glandium]]

Bug 1608037 is blocked on bug 1608026, which I haven't gotten back to figure out how to address. Bug 1427472 will help, though, and is more likely to be fixed soon.
Anyways, the bug as filed comes from user error, not from something having actually broken something else.