Closed
Bug 1234935
Opened 8 years ago
Closed 8 years ago
Thunderbird beta 43 build3 fails
Categories
(Thunderbird :: Build Config, defect)
Thunderbird
Build Config
Tracking
(Not tracked)
RESOLVED
FIXED
Thunderbird 46.0
People
(Reporter: wsmwk, Unassigned)
References
Details
Attachments
(1 file)
1.11 KB,
patch
|
Callek
:
review+
|
Details | Diff | Splinter Review |
cc: ted and bhearsum because jlund is thinking this is not an automation issue and suggested we consult some build experts. The relevant logs are cited below Thunderbird beta build3 has a new failure ... related to "repack_4/10 failed for Thunderbird 43.0b1 build3 on macosx64 Status: failure" jlund in tb-drivers: "good news is we have made it past previous error. bad news is we hit a new one on many of these failed repack jobs. looks like this time it may be a legit one and not at the fault of automation. failing to run 'make installers-{locale}' for certain locales: e.g. CalledProcessError: Command '['make', 'installers-gl']' returned non-zero exit status 2 Traceback (most recent call last): File "/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module> bucket_prefix=branchConfig['bucket_prefix'], File "/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks "Failed locales: %s" % " ".join([x for x, _ in failed])) Exception: Failed locales: fi fr fy-NL ga-IE gd gl **Full details are available at: http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/release-comm-beta-macosx64_repack_4%2F10/builds/5 -------------------------------- related to "repack_7/10 failed for Thunderbird 43.0b1 build3 on win32 Status: exception" jlund in tb-drivers: "the windows repack failures are different. looks like we are running into trouble diff'n a specific part. same last few lines on both win logs: patch-if "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}" "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calFilter.js.patch" "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calFilter.js" diffing "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calDuration.js" command timed out: 2400 seconds without output running ['bash', 'scripts/scripts/l10n/release_repacks.sh', 'win32', 'mozilla/production_config.py', '--chunks', '10', '--this-chunk', '7', '--stage-ssh-key', 'tbirdbld_dsa', '--stage-server', 'upload.tbirdbld.productdelivery.prod.mozaws.net', '--stage-username', 'tbirdbld', '--ftp-server', 'archive.mozilla.org', '--hghost', 'hg.mozilla.org', '--compare-locales-repo-path', 'build/compare-locales', '--use-pymake', '--generate-partials', '--tooltool-manifest', 'mail/config/tooltool-manifests/win32/releng.manifest', '--tooltool-script', 'python', '--tooltool-script', '/c/mozilla-build/tooltool.py', '--tooltool-url', 'https://api.pub.build.mozilla.org/tooltool/', '--balrog-api-root', 'https://aus4-admin.mozilla.org/api', '--balrog-username', 'tbirdbld', '--credentials-file', 'oauth.txt', '--bucket-prefix', 'net-mozaws-prod-delivery'], attempting to kill SIGKILL failed to kill process **Full details are available at: http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-comm-beta-win32_repack_7%2F10/builds/0
Comment 1•8 years ago
|
||
I'm not sure what's going on here. Might be worth asking Axel...looks like something is going wrong in the l10n build system.
Reporter | ||
Comment 2•8 years ago
|
||
(In reply to Ben Hearsum (:bhearsum) from comment #1) > I'm not sure what's going on here. Might be worth asking Axel...looks like > something is going wrong in the l10n build system.
Flags: needinfo?(l10n)
Comment 3•8 years ago
|
||
No idea, and I don't have access to the bb masters, so no access to the logs. Is this the first beta in the 43 cycle for tb? Also, why would this be a 43 instead of a 44?
Flags: needinfo?(l10n)
Reporter | ||
Comment 4•8 years ago
|
||
(In reply to Axel Hecht [:Pike] from comment #3) > No idea, and I don't have access to the bb masters, so no access to the logs. Public links are under https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/ Mac https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_4-bm86-build1-build5.txt.gz Windows https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/release-comm-beta-win32_repack_7-bm72-build1-build0.txt.gz > Is this the first beta in the 43 cycle for tb? It is the first beta aka b1, but the third build attempt because we've had numerous problems - automation, etc. > Also, why would this be a 43 instead of a 44? Because of the above, and our desire to use 44 beta for part 2 of the sha-1 fixes
Flags: needinfo?(l10n)
Comment 5•8 years ago
|
||
Given it's lproj, maybe something to match https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc is missing? With those, I'm really stumbling in the dark, maybe glandium has a better idea.
Flags: needinfo?(l10n)
Comment 6•8 years ago
|
||
(In reply to Axel Hecht [:Pike] from comment #5) > Given it's lproj, maybe something to match > https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc is missing? That was done in bug 1185268, so this should be in current comm-beta. But if it were that, it would fail for all locales, presumably. The thing to note, however, is that it works apparently once, and fails in subsequent runs, which, considering the failing command is a `mv`, can very well mean that there's something *not* happening in between the different locale runs that should be happening (because even before https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc , it was a `mv` as well).
Comment 7•8 years ago
|
||
Oh, those might actually be red herrings. The first real error seems to be: OSError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/comm-beta/mozilla/obj-l10n' Yay, one more /mozilla/ path issue.
Reporter | ||
Comment 8•8 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #7) > Oh, those might actually be red herrings. The first real error seems to be: > > OSError: [Errno 2] No such file or directory: > '/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/comm-beta/mozilla/obj-l10n' > > Yay, one more /mozilla/ path issue. Does this help nail it?
Flags: needinfo?(rail)
Flags: needinfo?(aleth)
Comment 9•8 years ago
|
||
Something tells me it may be related to unset MOZ_CURRENT_PROJECT... https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#184 tries to compare paths set in mozinfo.json and fails, because it skips https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#180
Flags: needinfo?(rail)
Comment 10•8 years ago
|
||
What can possibly go wrong?!
Attachment #8702974 -
Flags: review?(bugspam.Callek)
Comment 11•8 years ago
|
||
I pushed it remote: https://hg.mozilla.org/build/tools/rev/7d6cf1fdcf03 remote: https://hg.mozilla.org/build/tools/rev/2c636f6a0ec6 rerunning one of the repacks to see if it worked
Comment 12•8 years ago
|
||
Comment on attachment 8702974 [details] [diff] [review] tb_mac_repacks.diff if it works sure
Attachment #8702974 -
Flags: review?(bugspam.Callek) → review+
Comment 13•8 years ago
|
||
The trick above didn't work, so I played a bit on a builder and finally added a dirty hack to work around the /mozilla/ path issue: http://hg.mozilla.org/build/tools/file/a72c9646a5bd/scripts/l10n/create-release-repacks.py#l116 This is just a work around and should be addressed in a better way.
Comment 14•8 years ago
|
||
More fun: command: START command: c:\mozilla-build\python27\python.exe c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_7-000000\comm-beta/build/pymake/make.py --no-print-directory echo-variable-CHECKSUM_FILE AB_CD=pt-PT command: cwd: c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_7-000000\comm-beta\obj-l10n\mail\locales command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'pt-PT', 'MOZ_SIGN_CMD': 'python c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\signtool.py --cachedir c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\signing_cache -t c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\token -n c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\nonce -c c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'WIN32_INSTALLER_IN': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\thunderbird.exe', 'UPLOAD_SSH_KEY': '~\\.ssh\\tbirdbld_dsa', 'LOCALE_MERGEDIR': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\comm-beta\\\\\\\\obj-l10n\\\\\\\\mail\\\\\\\\locales\\\\\\\\merged', 'MOZILLA_OFFICIAL': '1', 'COMM_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/comm-beta/obj-l10n/dist/host/bin/mar.exe', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'ZIP_IN': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\thunderbird.zip', 'UPLOAD_USER': 'tbirdbld', 'USE_PYMAKE': '1', 'MBSDIFF': '/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/comm-beta/obj-l10n/dist/host/bin/mbsdiff.exe', 'UPLOAD_EXTRA_FILES': 'update/win32/pt-PT/thunderbird-42.0b1-43.0b1.partial.mar update/win32/pt-PT/thunderbird-42.0b1-43.0b1.partial.mar.asc update/win32/pt-PT/thunderbird-42.0b2-43.0b1.partial.mar update/win32/pt-PT/thunderbird-42.0b2-43.0b1.partial.mar.asc update/win32/pt-PT/thunderbird-41.0b2-43.0b1.partial.mar update/win32/pt-PT/thunderbird-41.0b2-43.0b1.partial.mar.asc', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 3 -v 43.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_PKG_VERSION': '43.0b1', 'MBSDIFF_HOOK': 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'} command: END (0.25s elapsed) command: output: '../../dist/win32/pt-PT//thunderbird-43.0b1.checksums' The following tracebacks were detected during repacks: nl: Traceback (most recent call last): File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/scripts/scripts/l10n/create-release-repacks.py", line 146, in createRepacks checksums = parseChecksumsFile(open(checksums_file).read()) IOError: [Errno 22] invalid mode ('r') or filename: "c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_7-000000\\comm-beta\\obj-l10n\\dist\\win32\\nl\\thunderbird-43.0b1.checksums'\r" Sounds like something has changed (on the builders?) and the command above uses CRLF. http://hg.mozilla.org/build/tools/rev/de42cac7de69 should fix it.
Reporter | ||
Comment 15•8 years ago
|
||
There were a few more Thunderbird 43.0b1 build3: failed at repack_6/10 on win32 11:16pm EST which succeeded two hours later Thunderbird 43.0b1 build3: completed repack_6/10 on win32 1:19am EST Then several of the type Thunderbird 43.0b1 build3: failed at update_verify_beta_2/6 on linux
Comment 16•8 years ago
|
||
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #15) > There were a few more > Thunderbird 43.0b1 build3: failed at repack_6/10 on win32 11:16pm EST > which succeeded two hours later > Thunderbird 43.0b1 build3: completed repack_6/10 on win32 1:19am EST > > Then several of the type > Thunderbird 43.0b1 build3: failed at update_verify_beta_2/6 on linux Those a different, bug 1209376
Reporter | ||
Comment 18•8 years ago
|
||
(In reply to aleth [:aleth] from comment #17) > Should this be resolved since the 43 beta shipped? depends on how rail wants to proceed, considering ... (In reply to Rail Aliiev [:rail] from comment #13) > The trick above didn't work, so I played a bit on a builder and finally > added a dirty hack to work around the /mozilla/ path issue: > http://hg.mozilla.org/build/tools/file/a72c9646a5bd/scripts/l10n/create- > release-repacks.py#l116 > > This is just a work around and should be addressed in a better way.
Assignee: nobody → rail
Flags: needinfo?(rail)
Comment 19•8 years ago
|
||
It works now, but the work around is very hacky. We may be hit again some time in the future again, unless we fix the root clause, which is somewhere in the build system AFAIK.
Assignee: rail → nobody
Flags: needinfo?(rail)
Comment 20•8 years ago
|
||
Resolving for now.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 46.0
You need to log in
before you can comment on or make changes to this bug.
Description
•