909952 - Thunderbird Windows L10n repack builds failed with pymake enabled

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Mark Banner (:standard8)]

Typical failure:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/candidates/24.0b1-candidates/build4/logs/release-comm-beta-win32_repack_2-bm58-build1-build0.txt.gz

e:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\comm-beta\obj-l10n\mail\locales\Makefile:98:0: command 'c:/mozilla-build/python27/python.exe e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/mozilla/build/pymake/pymake/../make.py repackage-zip \
  AB_CD=ca \
  MOZ_PKG_FORMAT=SFX7Z \
  ZIP_IN=e:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_2-000000\\thunderbird.exe \
  ZIP_OUT="e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/obj-l10n/mail/locales/../../mozilla/dist/win32/ca/Thunderbird Setup 24.0b1.exe" \
  SFX_HEADER="e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/obj-l10n/mail/locales/../installer/windows/l10ngen/7zSD.sfx \
              e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/mail/installer/windows/app.tag"' failed, return code 2
e:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\comm-beta\obj-l10n\mail\locales\Makefile:108:0: command 'c:/mozilla-build/python27/python.exe e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/mozilla/build/pymake/pymake/../make.py repackage-win32-installer AB_CD=ca WIN32_INSTALLER_IN=e:\\\\builds\\\\moz2_slave\\\\tb-rel-c-beta-w32_rpk_2-000000\\\\thunderbird.exe' failed, return code 2
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/mozilla/toolkit/mozapps/installer/l10n-repack.py", line 47, in <module>
    main()
  File "e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/comm-beta/mozilla/toolkit/mozapps/installer/l10n-repack.py", line 43, in main
    l10n.repack(args.build, args.l10n, args.non_resource, NON_CHROME)
  File "e:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\comm-beta\mozilla\python\mozbuild\mozpack\packager\l10n.py", line 172, in repack
    _repack(app_finder, l10n_finder, copier, formatter, non_chrome)
  File "e:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\comm-beta\mozilla\python\mozbuild\mozpack\packager\l10n.py", line 63, in _repack
    locale = app.locales[0]
IndexError: list index out of range
command: ERROR
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n\../../lib/python\util\commands.py", line 47, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'e:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_2-000000\\comm-beta/build/pymake/make.py', 'installers-ca']' returned non-zero exit status 1
command: END (26.71s elapsed)

Comment 1

[Rail Aliiev [:rail]]

ATM my staging win32 en-US build hits the following error:
e:\builds\moz2_slave\tb-rel-c-beta-w32_bld-00000000\build\config\rules.mk:1163:0$ e:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/objdir-tb/mozilla/_virtualenv/Scripts/python.exe -O e:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mozilla/build/cl.py cl -FoMapiDll.obj -c -D_HAS_EXCEPTIONS=0 -I../../../mozilla/dist/stl_wrappers  -DMOZILLA_INTERNAL_API -DMOZ_THUNDERBIRD=1 -DOSTYPE=\"WINNT6.1\" -DOSARCH=WINNT -DNO_NSPR_10_SUPPORT -DUNICODE -D_UNICODE  -Ie:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mailnews/mapi/mapiDLL -I. -I../../../mozilla/dist/include -I../../../mozilla/dist/include/nsprpub  -Ie:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/objdir-tb/mozilla/dist/include/nspr -Ie:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/objdir-tb/mozilla/dist/include/nss        -wd4099 -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4251 -wd4244 -wd4345 -wd4351 -wd4482 -wd4800 -wd4819 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -Zi -UDEBUG -DNDEBUG -O1 -Oy -MT            -FI ../../../mozilla/dist/include/mozilla-config.h -DMOZILLA_CLIENT e:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mailnews/mapi/mapiDLL/MapiDll.cpp
MapiDll.cpp

e:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mailnews/mapi/mapiDLL/MapiDll.cpp(7) : fatal error C1083: Cannot open include file: 'mapidefs.h': No such file or directory


Any idea why it may happen?

Comment 2

[Mark Banner (:standard8)]

That sounds like bug 869974 hasn't been deployed to the staging builder.

Comment 3

[Rail Aliiev [:rail]]

Yup installing the headers fixed it. Thanks for the tip.

Comment 4

[Rail Aliiev [:rail]]

Disabling pymake for l10n repacks made it work, but we still need to figure out how to build it with pymake.

http://hg.mozilla.org/build/buildbot-configs/rev/8349e2a30344

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #4)
> Disabling pymake for l10n repacks made it work, but we still need to figure
> out how to build it with pymake.
> 
> http://hg.mozilla.org/build/buildbot-configs/rev/8349e2a30344

Should this subject be updated to be more generic (ie, remove the "24b1" part), or is this bug fixed?

Comment 6

[Mark Banner (:standard8)]

Taking - will look at fixing this sometime over the next cycle

Comment 8

[[e-mail] (:bhearsum)]

The 28.0b1 release repacks failed, unsurprisingly, but they also didn't even try to use pymake. I think we need to flip  https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release-thunderbird-comm-beta.py#L97 around the time that this is fixed.

Comment 10

[Mark Banner (:standard8)]

I've been looking at this today, and from what I can tell, the big issue here is now that we're using the old msys make for the l10n repacks that now appears to be unsupported.

Switching to pymake shows encouraging signs, though I haven't yet been able to run it completely to be 100% sure enough it would work.

Comment 11

[Mark Banner (:standard8)]

Attachment: Use pymake for l10n builds as well

Here's the patch to enable pymake on l10n builds - Ben can you try this on staging please?

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Mark Banner (:standard8) from comment #11)
> Created attachment 8406880 [details] [diff] [review]
> Use pymake for l10n builds as well
> 
> Here's the patch to enable pymake on l10n builds - Ben can you try this on
> staging please?

http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-win32_repack_1%2F6/builds/7/steps/run_script/logs/stdio

command: START
command: c:\mozilla-build\python27\python.exe c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta/build/pymake/make.py -f client.mk configure
command: cwd: comm-beta
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'MOZ_SIGN_CMD': 'python c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\signtool.py --cachedir c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\signing_cache -t c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\token -n c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\nonce -c c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\host.cert -H signing6.srv.releng.scl3.mozilla.com:9110', 'UPLOAD_SSH_KEY': '~\\.ssh\\tbirdbld_dsa', 'MOZILLA_OFFICIAL': '1', 'COMM_REV': 'THUNDERBIRD_28_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'UPLOAD_USER': 'tbirdbld', 'USE_PYMAKE': '1', 'UPLOAD_HOST': 'dev-stage01.srv.releng.scl3.mozilla.com', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 1 -v 28.0b1 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'THUNDERBIRD_28_0b1_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '28.0b1'}
command: output:
Using MOZILLA_REV from environment (THUNDERBIRD_28_0b1_RELEASE)
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\mozilla']
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\mozilla']
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\util\commands.py", line 47, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 1
command: END (0.20s elapsed)

python: can't open file '../tools/buildfarm/utils/hgtool.py': [Errno 2] No such file or directory
python: can't open file '../tools/buildfarm/utils/hgtool.py': [Errno 2] No such file or directory
The exception was:
subprocess.CalledProcessError: Command '['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\mozilla']' returned non-zero exit status 2

Retrying previous command: 1 of 1 time(s)
The exception was:
subprocess.CalledProcessError: Command '['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\mozilla']' returned non-zero exit status 2

Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\client.py", line 656, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev, options.hgtool)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\client.py", line 334, in do_hg_pull
    check_call_noisy(['python'] + hgtoolcmd + [repository.rstrip('/'), fulldir], retryMax=options.retries)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\client.py", line 168, in check_call_noisy
    raise Exception("Command '%s' failed %d time(s). Giving up." % (cmd, retryMax + 1))
Exception: Command '['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\mozilla']' failed 2 time(s). Giving up.
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta/build/pymake/make.py", line 35, in <module>
    main(sys.argv[1:])
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta/build/pymake/make.py", line 27, in main
    clientpyargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\client.py', 'checkout', '--hg-options=--time', '--hgtool=../tools/buildfarm/utils/hgtool.py', '--skip-chatzilla', '--skip-comm', '--skip-inspector', '--skip-venkman', '--tinderbox-print']' returned non-zero exit status 1
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 342, in <module>
    buildid=options.buildid
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 98, in createRepacks
    makeDirs, localeSrcDir, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\build\l10n.py", line 86, in l10nRepackPrep
    env=env)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\util\commands.py", line 47, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 1

Comment 13

[Mark Banner (:standard8)]

Attachment: Debug for tools directory

Debug for the build tools, I'm not sure it'll tell us much though.

Comment 14

[Mark Banner (:standard8)]

Attachment: Debug args via client.mk

Debug for finding out what's being passed to client.py

Comment 15

[[e-mail] (:bhearsum)]

I've queued up a test with this at http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-win32_repack_1%2F6

Hopefully it will run overnight.

Comment 16

[Mark Banner (:standard8)]

Unfortuantely, that didn't work because of the change of repository name - something didn't pick up correctly.

If you need me to set up a user repo, let me know.

Comment 17

[[e-mail] (:bhearsum)]

We've been iterating on this for awhile

Comment 18

[Mark Banner (:standard8)]

Comment on attachment 8406880 [details] [diff] [review]
Use pymake for l10n builds as well

In combination with a few other non-buildbot patches, I've now had a staging run complete successfully with Lightning disabled.

Therefore this patch is definitely needed.

The Lightning patch I'll confirm if it is required or not later. I'll also get the other patches up later.

Comment 19

[Mark Banner (:standard8)]

First step:

Landed Port of bug 741014 - https://hg.mozilla.org/comm-central/rev/494eb7379715 onto comm-central.

Comment 20

[Mark Banner (:standard8)]

Second step: I've just set off a new staging build with a few possible improvements - Ben, can we get a builder hooked up again when you're about (they seem to be offline on that master atm).

Comment 21

[Mark Banner (:standard8)]

(In reply to Mark Banner (:standard8) (away until 22 Apr) from comment #19)
> First step:
> 
> Landed Port of bug 741014 -
> https://hg.mozilla.org/comm-central/rev/494eb7379715 onto comm-central.

Landed on branches in preparation:

https://hg.mozilla.org/releases/comm-aurora/rev/dfac43769e7a
https://hg.mozilla.org/releases/comm-beta/rev/ff5101cc8e05

Comment 22

[[e-mail] (:bhearsum)]

(In reply to Mark Banner (:standard8) (away until 22 Apr) from comment #20)
> Second step: I've just set off a new staging build with a few possible
> improvements - Ben, can we get a builder hooked up again when you're about
> (they seem to be offline on that master atm).

http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-win32_repack_1%2F6/builds/25

Comment 23

[Mark Banner (:standard8)]

Attachment: Workaround client.py / hgtool options

There's two parts in here. One is to drop MOZILLA_REV and COMM_REV from the l10n args file - these are specified in the environment and picked up by hgtool, and don't actually get picked up correctly when client.py is loaded from the c-c pymake.

The second part enables hgtool for l10n repacks, cutting down repack times - the issue is that l10n builds have a different location for the scripts, hence the additional arguments.

We could probably de-duplicate the args files, but I thought we'd leave it for now.

Comment 24

[Mark Banner (:standard8)]

Comment on attachment 8406880 [details] [diff] [review]
Use pymake for l10n builds as well

https://hg.mozilla.org/build/buildbot-configs/rev/49cd9523e988

Comment 25

[Mark Banner (:standard8)]

Comment on attachment 8409797 [details] [diff] [review]
Workaround client.py / hgtool options

https://hg.mozilla.org/comm-central/rev/4409679b6df5

Comment 26

[Mark Banner (:standard8)]

Comment on attachment 8409797 [details] [diff] [review]
Workaround client.py / hgtool options

Also landed on branches with a=me:

https://hg.mozilla.org/releases/comm-aurora/rev/c82a2bbf3839
https://hg.mozilla.org/releases/comm-beta/rev/e55c8235cba0

Comment 27

[Mark Banner (:standard8)]

Comment on attachment 8406880 [details] [diff] [review]
Use pymake for l10n builds as well

Also landed on production with agreement from bhearsum:

https://hg.mozilla.org/build/buildbot-configs/rev/bb24b0ee8dc2

Comment 28

[Mark Banner (:standard8)]

This should be fixed now.

Comment 29

[Jordan Lund (:jlund)]

buildbot-config patch live in production: http://hg.mozilla.org/build/buildbot-configs/rev/5fce6d67a084 :)