1323107 - thunderbird version 51 beta, all win32 repacks fail

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Wayne Mery (:wsmwk)]

+++ This bug was initially created as a clone of Bug #1321994 +++
+++ This bug was initially created as a clone of Bug #1317863 +++

all win32 repacks failed.  linux and Mac succeed.

rail	wsmwk: not sure what is wrong, the following command exits 2 with no output:
rail	c:\mozilla-build\python27\python.exe c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_8-000000\comm-beta/build/pymake/make.py -f client.mk configure
rail	maybe the file (make.py) doesn't exist? 

from https://ftp.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build4/logs/release-comm-beta-linux_repack_8-bm72-build1-build4.txt.gz

Comment 1

[aleth [:aleth]]

In a different log, the error actually happens in client.mk:

command: START
command: c:\mozilla-build\python27\python.exe c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta/build/pymake/make.py -f client.mk configure
command: cwd: comm-beta
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'MOZ_SIGN_CMD': 'python c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\signtool.py --cachedir c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\signing_cache -t c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\token -n c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\nonce -c c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_1-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~\\.ssh\\tbirdbld_dsa', 'MOZILLA_OFFICIAL': '1', 'COMM_REV': 'THUNDERBIRD_51_0b1_RELEASE', 'MOZ_OBJDIR': 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta\\obj-l10n', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'UPLOAD_USER': 'tbirdbld', 'USE_PYMAKE': '1', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 4 -v 51.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_51_0b1_RELEASE', 'MOZ_PKG_VERSION': '51.0b1', 'MBSDIFF_HOOK': 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'}
command: output:
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
command: END (1.56s elapsed)

client.mk:192: *** multiple target patterns.  Stop.
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\build\l10n.py", line 85, in l10nRepackPrep
    env=env)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_1-000000\\comm-beta/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
program finished with exit code 1
elapsedTime=115.172000

https://ftp.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build4/logs/release-comm-beta-win32_repack_1-bm72-build1-build14.txt.gz

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to aleth [:aleth] from comment #1)
> In a different log, the error actually happens in client.mk:

Sorry, the correct URL for comment 0 is https://ftp.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build4/logs/release-comm-beta-win32_repack_8-bm73-build1-build2.txt.gz which shows the same error 

Solution?

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The error seems to indicate there is a colon in one of the variable assignments where it shouldn't be (or should be quoted). The mentioned bucket_prefix is the last argument, so it might be also one of the earlier ones, because
> --bucket-prefix net-mozaws-prod-delivery
looks normal.

https://github.com/mozilla/build-tools/blob/master/scripts/l10n/create-release-repacks.py#L394

Comment 4

[Joe Sabash [:JoeS1]]

Similar error now when attempting repacks for TB 45.6.0
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-esr45-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr45-w32_rpk_1-00000\\comm-esr45/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
command: END (2.62s elapsed)

client.mk:199: *** multiple target patterns.  Stop.
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-esr45-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-esr45-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-esr45-w32_rpk_1-00000\scripts\lib\python\build\l10n.py", line 85, in l10nRepackPrep
    env=env)
  File "c:\builds\moz2_slave\tb-rel-c-esr45-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr45-w32_rpk_1-00000\\comm-esr45/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
program finished with exit code 1

Comment 5

[Wayne Mery (:wsmwk)]

indeed. and only windows as per this bug

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #3)
> The error seems to indicate there is a colon in one of the variable
> assignments where it shouldn't be (or should be quoted). The mentioned
> bucket_prefix is the last argument, so it might be also one of the earlier
> ones, because
> > --bucket-prefix net-mozaws-prod-delivery
> looks normal.
> 
> https://github.com/mozilla/build-tools/blob/master/scripts/l10n/create-
> release-repacks.py#L394

aleth, does that help?

Comment 7

[aleth [:aleth]]

It seems make interprets the : in Windows paths (C:\...) as beginning another target definition, hence the error.

Comment 8

[aleth [:aleth]]

Given http://searchfox.org/comm-central/source/client.mk#155, I'm surprised client.mk doesn't like the absolute MOZ_OBJDIR on Windows (always assuming that's the change that caused this bustage).

I'm not sure how to interpret the line number 192?

Comment 9

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #8)
> Given http://searchfox.org/comm-central/source/client.mk#155, I'm surprised
> client.mk doesn't like the absolute MOZ_OBJDIR on Windows (always assuming
> that's the change that caused this bustage).

That code block is a little confusing though...

Comment 10

[Wayne Mery (:wsmwk)]

rail, does the above, and the following, now help point to the cause?

Aryx	line 158 looks like it doesn't support absolute paths: http://searchfox.org/comm-central/source/client.mk#158
aleth	Aryx: exactly. But... why does it work on Linux and OS X?
Aryx	they already have /foo and not c:\...?
aleth	But they also get passed an absolute path and the join should produce nonsense there too
aleth	which makes me wonder if that code block even gets executed

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps you should back out https://hg.mozilla.org/build/tools/rev/633092dc002e, as it's broken 45.6.0 too. Then pursue a different fix for bug 1317863.

Comment 12

[Rail Aliiev [:rail]]

I think Nick is right, we should try and backout https://hg.mozilla.org/build/tools/rev/633092dc002e

Comment 13

[aleth [:aleth]]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #12)
> I think Nick is right, we should try and backout
> https://hg.mozilla.org/build/tools/rev/633092dc002e

Yes, that will unbreak 45.6.0 at least, and there might be a more targeted way to fix bug 1317863.

Comment 14

[aleth [:aleth]]

Resolved as per comment 11.