1297785 - Thunderbird 49.0b1 build4 fails on tooltool for windows repacks

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Wayne Mery (:wsmwk)]

from bug 1286440 comment 91

ERROR - The following files failed: 'vs2015u2.zip'
retry: Giving up on <function run_with_timeout at 0x02831C70>
Unable to successfully run ['python.exe', 'c:/mozilla-build/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch'] after 10 attempts
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
command: END (1747.77s elapsed)

Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\build\l10n.py", line 80, in l10nRepackPrep
    run_cmd(cmd, cwd=sourceRepoName)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
program finished with exit code 1
elapsedTime=1906.894000

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like the vs2015u2.zip file is marked internal-only in tooltool, so the auth token needs to be passed to the tooltool script.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Pass the auth file

Quick'n'dirty fix. Hopefully it has the right number of \

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8785221 [details] [diff] [review]
[buildbotcustom] Pass the auth file

Review of attachment 8785221 [details] [diff] [review]:
-----------------------------------------------------------------

this will be fun seeing how well these slashes hold up as they go through py -> sh -> py on windows :)

I'm wondering if we should have buildbot handle the tooltool downloads[1] since it seems to always pass the auth file location[2]. In otherwords, remove the tooltool args and pass SigningScriptFactory `tooltool_manifest_src` in args. On second thought, that seems way to invasive :P

[1] https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4487
[2] https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#387

::: process/release.py
@@ +717,5 @@
>                      extra_args.extend(['--tooltool-manifest',
>                                        pf.get('tooltool_manifest_src')])
>                  if pf.get('tooltool_script'):
>                      for script in pf['tooltool_script']:
>                          extra_args.extend(['--tooltool-script', script])

at first I thought your clipboard was broken with all of those --tooltool-script args..

@@ +718,5 @@
>                                        pf.get('tooltool_manifest_src')])
>                  if pf.get('tooltool_script'):
>                      for script in pf['tooltool_script']:
>                          extra_args.extend(['--tooltool-script', script])
> +                    if platform == 'win32':

should we future proof this and add win64? Or could that cause spurious results? :)

I'm fine with restricting scope to this bug and tb

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8785221 [details] [diff] [review]
[buildbotcustom] Pass the auth file

I included the suggestion about win64 when landing:
  https://hg.mozilla.org/build/buildbotcustom/rev/8d101bcc62ce
  https://hg.mozilla.org/build/buildbotcustom/rev/47647a3fb465

Another option to improve this would be to have an arg for create-release-repacks.py for the tooltook auth token, like we have for manifest, url, and script. Still wouldn't let us leverage the ToolToolMixin though.

Leaving open to verify there are enough \\.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Fixed in 49.0b1 build5:

command: START
command: sh ../scripts/scripts/tooltool/tooltool_wrapper.sh mail/config/tooltool-manifests/win32/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh python c:/mozilla-build/tooltool.py --authentication-file c:\builds\relengapi.tok
command: cwd: comm-beta
command: output:
mail/config/tooltool-manifests/win32/releng.manifest content
=======================================================
[
{
"size": 266240,
"digest": "bb345b0e700ffab4d09436981f14b5de84da55a3f18a7f09ebc4364a4488acdeab8d46f447b12ac70f2da1444a68b8ce8b8675f0dae2ccf845e966d1df0f0869",
"algorithm": "sha512",
"filename": "mozmake.exe"
},
{
"version": "rustc 1.9.0 (e4e8b6668 2016-05-18)",
"size": 82463178,
"digest": "a3c54c6792e75d53ec79caf958db25b651fcf968a37b00949fb327c54a54cad6305a4af302f267082d86d70fcf837ed0f273f85b97706c20b957ff3690889b40",
"algorithm": "sha512",
"filename": "rustc.tar.bz2",
"unpack": true
},
{
"size": 167175,
"digest": "0b71a936edf5bd70cf274aaa5d7abc8f77fe8e7b5593a208f805cc9436fac646b9c4f0b43c2b10de63ff3da671497d35536077ecbc72dba7f8159a38b580f831",
"algorithm": "sha512",
"filename": "sccache.tar.bz2",
"unpack": true
},
{
"version": "Visual Studio 2015 Update 2 / SDK 10.0.10586.0/212",
"size": 332343834,
"digest": "55814aaabcd4aa51fe85918ec02a8c29bc067d41ee79ddcfd628daaba5a06d4241a73a51bf5a8bc69cc762b52551009f44b05e65682c45b4684c17fb2d017c2c",
"algorithm": "sha512",
"filename": "vs2015u2.zip",
"unpack": true
}
]
=======================================================
Fetching...
Executing: ['python.exe', 'c:/mozilla-build/tooltool.py', '--authentication-file', 'c:\\builds\\relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch']
INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
INFO - File mozmake.exe fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\tmpub1rcn
INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
INFO - File rustc.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\tmp9fhfkc
INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
INFO - File sccache.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\tmpmbf_yq
INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
INFO - File vs2015u2.zip fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_1-000000\comm-beta\tmpdglnyi
INFO - File integrity verified, renaming tmpub1rcn to mozmake.exe
INFO - File integrity verified, renaming tmp9fhfkc to rustc.tar.bz2
INFO - File integrity verified, renaming tmpmbf_yq to sccache.tar.bz2
INFO - File integrity verified, renaming tmpdglnyi to vs2015u2.zip
INFO - untarring "rustc.tar.bz2"
INFO - untarring "sccache.tar.bz2"
INFO - unzipping "vs2015u2.zip"
command: END (49.50s elapsed)