1594682 - Perma esr68 win2012 mozmake.exe[2]: *** [repackage-zip] Error 1 | make installers-en-CA failed

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=274641974&repo=mozilla-esr68
Full log: https://queue.taskcluster.net/v1/task/NNu6BcXHS6ObYX02N3BqvQ/runs/0/artifacts/public/logs/live_backing.log

---

15:37:51 INFO - processing z:/task_1572965515/build/src/browser/branding/official/locales/jar.mn
15:37:51 INFO - adding 'browser' entry to root chrome manifest appid={ec8030f7-c20a-464f-9b0e-13a3a9e97384}
15:37:51 INFO - mozmake.exe[3]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/branding/official/locales'
15:37:51 INFO - mozmake.exe[3]: Entering directory 'z:/task_1572965515/build/src/obj-firefox/browser/branding/official/locales'
15:37:51 INFO - mozmake.exe[3]: Nothing to be done for 'tools'.
15:37:51 INFO - mozmake.exe[3]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/branding/official/locales'
15:37:51 INFO - mozmake.exe[2]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/branding/official/locales'
15:37:51 INFO - mozmake.exe[1]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:51 INFO - mozmake.exe[1]: Entering directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:51 INFO - z:/task_1572965515/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/task_1572965515/build/src/config/nsinstall.py -D ../../dist/win64/xpi/
15:37:51 INFO - z:/task_1572965515/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe -m mozbuild.action.langpack_manifest --locales en-CA --min-app-ver 68.3.0 --max-app-ver 68.* --app-name 'Firefox' --l10n-basedir 'z:/task_1572965515/build/l10n' --defines z:/task_1572965515/build/l10n/en-CA/toolkit/defines.inc z:/task_1572965515/build/l10n/en-CA/browser/defines.inc --langpack-eid '[e-mail]' --input ../../dist/xpi-stage/locale-en-CA
15:37:52 INFO - z:/task_1572965515/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe -m mozbuild.action.zip -C ../../dist/xpi-stage/locale-en-CA -x **/.manifest -x **/.js -x **/*.ini z:/task_1572965515/build/src/obj-firefox/dist/win64/xpi/firefox-68.3.0.en-CA.langpack.xpi chrome localization browser manifest.json
15:37:52 INFO - mozmake.exe[1]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:52 INFO - mozmake.exe[1]: Entering directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:52 INFO - rm -f -r -f z:/task_1572965515/build/src/obj-firefox/dist/l10n-stage
15:37:52 INFO - z:/task_1572965515/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/task_1572965515/build/src/config/nsinstall.py -D z:/task_1572965515/build/src/obj-firefox/dist/l10n-stage
15:37:52 INFO - (cd z:/task_1572965515/build/src/obj-firefox/dist/unpacked-installer/. && C:/mozilla-build/msys/bin/tar.exe --exclude=.mkdir.done -chf - .) | (cd z:/task_1572965515/build/src/obj-firefox/dist/l10n-stage/. && tar -xf -)
15:37:53 INFO - mozmake.exe[2]: Entering directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:53 INFO - z:/task_1572965515/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/task_1572965515/build/src/toolkit/mozapps/installer/l10n-repack.py 'z:/task_1572965515/build/src/obj-firefox/dist/l10n-stage/firefox' ../../dist/xpi-stage/locale-en-CA
15:37:53 INFO -
15:37:53 ERROR - Traceback (most recent call last):
15:37:53 INFO - File "z:/task_1572965515/build/src/toolkit/mozapps/installer/l10n-repack.py", line 58, in <module>
15:37:53 INFO - main()
15:37:53 INFO - File "z:/task_1572965515/build/src/toolkit/mozapps/installer/l10n-repack.py", line 54, in main
15:37:53 INFO - non_resources=args.non_resource, non_chrome=NON_CHROME)
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\packager\l10n.py", line 286, in repack
15:37:53 INFO - app_finder = UnpackFinder(source)
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\packager\unpack.py", line 76, in init
15:37:53 INFO - self._fill_with_jar(p[:-len(self.omnijar) - 1], jar)
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\packager\unpack.py", line 103, in _fill_with_jar
15:37:53 INFO - for e in parse_manifest(None, path, j):
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\chrome\manifest.py", line 361, in parse_manifest
15:37:53 INFO - e = parse_manifest_line(base, line)
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\chrome\manifest.py", line 344, in parse_manifest_line
15:37:53 INFO - return errors.fatal('Unknown manifest directive: %s' % cmd[0])
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\errors.py", line 103, in fatal
15:37:53 INFO - self._handle(self.FATAL, msg)
15:37:53 INFO - File "z:\task_1572965515\build\src\python\mozbuild\mozpack\errors.py", line 98, in _handle
15:37:53 INFO - raise ErrorMessage(msg)
15:37:53 INFO - mozpack.errors.ErrorMessage: Error: browser/META-INF/cose.manifest:1: Unknown manifest directive: Manifest-Version:
15:37:53 INFO - z:/task_1572965515/build/src/toolkit/locales/l10n.mk:115: recipe for target 'repackage-zip' failed
15:37:53 INFO - mozmake.exe[2]: *** [repackage-zip] Error 1
15:37:53 INFO - mozmake.exe[2]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:53 INFO - z:/task_1572965515/build/src/toolkit/locales/l10n.mk:155: recipe for target 'repackage-zip-en-CA' failed
15:37:53 INFO - mozmake.exe[1]: *** [repackage-zip-en-CA] Error 2
15:37:53 INFO - mozmake.exe[1]: Leaving directory 'z:/task_1572965515/build/src/obj-firefox/browser/locales'
15:37:53 INFO - Makefile:109: recipe for target 'installers-en-CA' failed
15:37:53 INFO - mozmake.exe: *** [installers-en-CA] Error 2
15:37:53 ERROR - Return code: 2
15:37:53 ERROR - make installers-en-CA failed
15:37:53 ERROR - failure: repack_locale(en-CA)
15:37:53 ERROR - # TBPL FAILURE #

Comment 1

[Andreea Pavel [:apavel]]

Mihai, please take a look.

Comment 2

[Andreea Pavel [:apavel]]

Justin please take a look.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

FYI Callek: this ocurred while I was still uplifting patches to esr68 on Monday. It failed on this specific revision but then went green after I uplifted:

• https://hg.mozilla.org/releases/mozilla-esr68/rev/928b6280acdecf06ade06c6554312ef9ba48b77e
• https://hg.mozilla.org/releases/mozilla-esr68/rev/f7cdd753769f223fe8a55017cd55a48f2c46eeab
• https://hg.mozilla.org/releases/mozilla-esr68/rev/dbb39be653579ce735fb0dceeeb7ad51fc3c153e

Neither of the patches are related though with l10n package, but more with signing.
Reruns didn't work. Could'be been intermitenttly something else?

Comment 4

[Justin Wood (:Callek)]

I just triggered win64's L10n job on that following push and the prior one to this....

I have a (few) minor theories, but lets see what this shows first... https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&searchStr=l10n&revision=dbb39be653579ce735fb0dceeeb7ad51fc3c153e and https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&searchStr=l10n&revision=3bade4adecfdb654e2a912355b0627365b681551

Comment 5

[Justin Wood (:Callek)]

For my reference, Bug 1533818 is probably what we need

Comment 6

[Justin Wood (:Callek)]

Ok, yea https://phabricator.services.mozilla.com/D34697 needs to be grafted to esr...

Comment 7

[Intermittent Failures Robot]

5 failures in 4850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 5

Platform breakdown:

• windows2012-64-shippable: 4
• windows2012-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1594682&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 8

[Jordan Lund (:jlund)]

(In reply to Justin Wood (:Callek) from comment #6)

Ok, yea https://phabricator.services.mozilla.com/D34697 needs to be grafted to esr...

Looks like your patch. could you uplift it? Tom is going to look into uplifting all things for esr68 but since this is busted now, it be great to resolve.

Comment 9

[Andreea Pavel [:apavel]]

(In reply to Jordan Lund (:jlund) from comment #8)

(In reply to Justin Wood (:Callek) from comment #6)

Ok, yea https://phabricator.services.mozilla.com/D34697 needs to be grafted to esr...

Looks like your patch. could you uplift it? Tom is going to look into uplifting all things for esr68 but since this is busted now, it be great to resolve.

Hi Jordan, i tried to graft that but it didn't apply cleanly.
Got the following:

mozilla@ubuntu ~/mozilla-unified esr68(0) $ hg graft -er 69128ae35322::2511377ca37f
grafting 545976:69128ae35322 "Bug 1533818 - Start signing omni.ja files. r=aki"
merging taskcluster/taskgraph/util/signed_artifacts.py
warning: conflicts while merging taskcluster/taskgraph/util/signed_artifacts.py! (edit, then use 'hg resolve --mark')
abort: unresolved conflicts, can't continue
(use 'hg resolve' and 'hg graft --continue')
mozilla@ubuntu ~/mozilla-unified esr68(0+changes) $

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Only 2511377ca37f was needed on ESR68 and has been uplifted to both 68.3.0 an 68.2.x.

Comment 11

[Justin Wood (:Callek)]

https://hg.mozilla.org/releases/mozilla-esr68/rev/bee37ce71ba5025366206c031f82f559c2d02382
https://hg.mozilla.org/releases/mozilla-esr68/rev/995b67ef37a9f67687d786e5ce1bb244d2e76b99

Comment 12

[Intermittent Failures Robot]

6 failures in 4207 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 6

Platform breakdown:

• osx-shippable: 2
• linux32-shippable: 1
• windows2012-32-shippable: 1
• windows2012-64-shippable: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1594682&startday=2019-11-11&endday=2019-11-17&tree=all