1231174 - Thunderbird 43.0b1 repacks failing on mac

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

All 10 chunks failed in packaging, eg 

http://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build1/logs/release-comm-beta-macosx64_repack_9-bm84-build1-build5.txt.gz:

cd ../../dist/l10n-stage; \
	  (cd thunderbird/Thunderbird.app/Contents/Resources && /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n/_virtualenv/bin/python /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/config/createprecomplete.py) && cd ./thunderbird && python /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/scripts/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/signing_cache -t /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/token -n /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/nonce -c /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 -f dmgv2   Thunderbird.app && cd /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n/dist/l10n-stage && /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n/_virtualenv/bin/python -m mozbuild.action.make_dmg 'thunderbird' 'mac/rm/Thunderbird 43.0b1.dmg'
2015-12-07 16:33:48,485 - ce8abe42eaffbaf1189636d4aea814c1a2275a65: processing Thunderbird.app.tar.gz on https://mac-v2-signing7.srv.releng.scl3.mozilla.com:9120
2015-12-07 16:33:48,617 - ce8abe42eaffbaf1189636d4aea814c1a2275a65: uploading for signing
2015-12-07 16:33:58,878 - ce8abe42eaffbaf1189636d4aea814c1a2275a65: processing Thunderbird.app.tar.gz on https://mac-v2-signing7.srv.releng.scl3.mozilla.com:9120
2015-12-07 16:34:35,151 - ce8abe42eaffbaf1189636d4aea814c1a2275a65: OK
2015-12-07 16:34:35,151 - Copying Thunderbird.app.tar.gz to cache /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/signing_cache/dmg/ce8abe42eaffbaf1189636d4aea814c1a2275a65
Traceback (most recent call last):
  File "/tools/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tools/python/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 37, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 32, in main
    make_dmg(args[0], args[1])
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 15, in make_dmg
    build = MozbuildObject.from_environment()
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/python/mozbuild/mozbuild/base.py", line 184, in from_environment
    if not samepath(topobjdir, _config_topobjdir):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/python/mozbuild/mozbuild/base.py", line 46, in samepath
    return os.path.samefile(path1, path2)
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n/_virtualenv/lib/python2.7/posixpath.py", line 154, in samefile
    s2 = os.stat(f2)
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/obj-l10n'
make[1]: *** [repackage-zip] Error 255
make: *** [repackage-zip-rm] Error 2
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_8-000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', 'installers-rm']' returned non-zero exit status 2

There is a 
  /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n
but not a 
  /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/obj-l10n

Anyone know of changes which might be causing this on mac, but not on the other platforms ?

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> All 10 chunks failed in packaging, eg 
> 
> http://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/
> build1/logs/release-comm-beta-macosx64_repack_9-bm84-build1-build5.txt.gz:
> ...
> 
> There is a 
>   /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/obj-l10n
> but not a 
>   /builds/slave/tb-rel-c-beta-m64_rpk_8-000000/comm-beta/mozilla/obj-l10n
> 
> Anyone know of changes which might be causing this on mac, but not on the
> other platforms ?

This is still during the signing piece?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I think that completed, and it's trying to package the dmg. A 42.0b1 log has this next:

mktemp -d mac/sl/pkg-dmg.12641.XXXXXXXX
mac/sl/pkg-dmg.12641.hOVahD3i
mkdir mac/sl/pkg-dmg.12641.hOVahD3i/stage
rsync -a --copy-unsafe-links thunderbird/ mac/sl/pkg-dmg.12641.hOVahD3i/stage
mkdir -p mac/sl/pkg-dmg.12641.hOVahD3i/stage/.background
rsync -a --copy-unsafe-links /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/dist/branding/dsstore mac/sl/pkg-dmg.12641.hOVahD3i/stage/.DS_Store
rsync -a --copy-unsafe-links /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/dist/branding/background.png mac/sl/pkg-dmg.12641.hOVahD3i/stage/.background
...

and note the references make_dmg.py. I think the call is made here
http://hg.mozilla.org/releases/mozilla-beta/file/THUNDERBIRD_43_0b1_RELEASE/toolkit/mozapps/installer/upload-files.mk#l529

Comment 3

[Wayne Mery (:wsmwk)]

> 529 INNER_MAKE_PACKAGE	= $(call py_action,make_dmg,'$(PKG_DMG_SOURCE)' '$(PACKAGE)')

Thoughts?

Comment 4

[Rail Aliiev [:rail]]

I am not that familiar with peace of the build system, but there are some ideas:

* make_dmg() calls MozbuildObject.from_environment() in https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/make_dmg.py#15

* MozbuildObject.from_environment()'s cwd is /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/dist/l10n-stage (from the log)

* it tries to detect topsrcdir walking ancestor directories searching either for mozinfo.json or python/mozbuild/mozbuild/base.py 

* it finds mozinfo.json in /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n and tries to read it:

{"pgo": false, "toolkit": "cocoa", "webm": true, "buildapp": "../mail", "crashreporter": true, "addon_signing": false, "platform_guess": null, "appname": "thunderbird", "mozconfig": "/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/.mozconfig", "topsrcdir": "/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/mozilla", "telemetry": false, "datareporting": true, "buildtype_guess": "opt", "bits": 64, "bin_suffix": "", "wave": true, "healthreport": false, "release_build": true, "asan": false, "tests_enabled": true, "official": false, "tsan": false, "debug": false, "os": "mac", "processor": "x86_64"}

* topsrcdir is set to /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/mozilla in that file

* then it does some magic to calculate _config_topobjdir 

I bet there is a way to make it work by passing MOZ_CURRENT_PROJECT, but it'd be better to ask the Build Config folks, see https://wiki.mozilla.org/Modules/Core#Build_Config

Comment 5

[Wayne Mery (:wsmwk)]

Please see comment 4.
This is blocking shipping our beta builds.
Thanks

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/build/buildbotcustom/rev/032b787e4dfe took care of the buildbot cases (eg dep and release en-US builds). The repacks use a separate script, so it's possible that we need another special case here:
http://hg.mozilla.org/build/tools/file/e3857f4deaba/lib/python/build/l10n.py#l147

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Specifically there's no MOZ_CURRENT_PROJECT in the env when make installers-%s is called at http://hg.mozilla.org/build/tools/file/438be5ffd006/lib/python/build/l10n.py#l188   (NB used an old rev by mistake at the end of comment #6).

command: START
command: make installers-sl
command: cwd: /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/mail/locales
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'sl', 'MOZ_SIGN_CMD': 'python /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/scripts/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/signing_cache -t /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/token -n /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/nonce -c /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/tbirdbld_dsa', 'LOCALE_MERGEDIR': '/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/mail/locales/merged', 'MOZ_PKG_PLATFORM': 'mac', 'COMM_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/dist/host/bin/mar', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'ZIP_IN': '/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/thunderbird.dmg', 'UPLOAD_USER': 'tbirdbld', 'MBSDIFF': '/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/comm-beta/obj-l10n/dist/host/bin/mbsdiff', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 1 -v 43.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_PKG_VERSION': '43.0b1', 'MBSDIFF_HOOK': '/builds/slave/tb-rel-c-beta-m64_rpk_9-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /builds/slave/tb-rel-c-beta-m64_rpk_9-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'}

The strange thing is that Firefox doesn't hit this problem.

Comment 8

[Mike Hommey [:glandium]]

Related bugs: bug 1191598, bug 1220018, bug 1220763.

MOZ_CURRENT_PROJECT should already be passed around.

Comment 9

[Wayne Mery (:wsmwk)]

question for anyone, but NI nick:

 - The merge for Thunderbird branches has been done such that 44 is theoretically ready.  But if we fix this bug, is it correct that the remainder of the 43 beta process be run, and we can then ship version 43 beta?    (I hope so - because I do not foresee us having time to get 44 beta ready in the next week)

- can you look at bug 1214403 to see if it will help this bug?  (fallen thinks it may help)

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

You can still complete 43.0b1 so long as 44.0b1 has not been started in ship-it. The release automation will use the THUNDERBIRD_43_0b1_RELEASE tag to pull the older code.

I don't have time to wrap my head around why this is broken for Thunderbird and not for Firefox. AFAICT this isn't a universal build so I don't follow why bug 1214403/MOZ_CURRENT_PROJECT are related. Fallen, do you have more context ?

Comment 11

[Wayne Mery (:wsmwk)]

Attachment: patch for Thunderbird Mac repack problem

automation patch by fallen
reviewed by rail
hopefully to be checked in to https://hg.mozilla.org/build/buildbotcustom/ by patrick :)

ref bug 1220018

Comment 12

[Patrick Cloke [:clokep]]

Comment on attachment 8700030 [details] [diff] [review]
patch for Thunderbird Mac repack problem

I think this was already looked at and I was asked to push this through the review process. I didn't see any approval for this patch, so sorry if you've reviewed this already!

Comment 13

[Patrick Cloke [:clokep]]

Thank you for the quick review! I pushed this as https://hg.mozilla.org/build/buildbotcustom/rev/7bef77bbaff8

Comment 14

[Patrick Cloke [:clokep]]

I'm going to resolve this as fixed. This was merged into production in https://hg.mozilla.org/build/buildbotcustom/changeset/98e9120ade54

Not sure if reconfigs have happened yet.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to Patrick Cloke [:clokep] from comment #14) 
> Not sure if reconfigs have happened yet.

They did according to rail. 

43.b1 build2 started, but is failing at linux mach. Perhaps bug 1233577