Closed Bug 1234935 Opened 8 years ago Closed 8 years ago

Thunderbird beta 43 build3 fails

Categories

(Thunderbird :: Build Config, defect)

defect
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 46.0

People

(Reporter: wsmwk, Unassigned)

References

Details

Attachments

(1 file)

cc: ted and bhearsum because jlund is thinking this is not an automation issue and suggested we consult some build experts. The relevant logs are cited below

Thunderbird beta build3 has a new failure ...


related to "repack_4/10 failed for Thunderbird 43.0b1 build3 on macosx64
Status: failure"

jlund in tb-drivers: "good news is we have made it past previous error. bad news is we hit a new one on many of these failed repack jobs. looks like this time it may be a legit one and not at the fault of automation. failing to run 'make installers-{locale}' for certain locales:

e.g. CalledProcessError: Command '['make', 'installers-gl']' returned non-zero exit status 2

Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks
    "Failed locales: %s" % " ".join([x for x, _ in failed]))
Exception: Failed locales: fi fr fy-NL ga-IE gd gl

**Full details are available at:
 http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/release-comm-beta-macosx64_repack_4%2F10/builds/5

--------------------------------

related to "repack_7/10 failed for Thunderbird 43.0b1 build3 on win32
Status: exception"

jlund in tb-drivers: "the windows repack failures are different. looks like we are running into trouble diff'n a specific part. same last few lines on both win logs:

   patch-if "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}" "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calFilter.js.patch" "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calFilter.js"
diffing "distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/calendar-js/calDuration.js"

command timed out: 2400 seconds without output running ['bash', 'scripts/scripts/l10n/release_repacks.sh', 'win32', 'mozilla/production_config.py', '--chunks', '10', '--this-chunk', '7', '--stage-ssh-key', 'tbirdbld_dsa', '--stage-server', 'upload.tbirdbld.productdelivery.prod.mozaws.net', '--stage-username', 'tbirdbld', '--ftp-server', 'archive.mozilla.org', '--hghost', 'hg.mozilla.org', '--compare-locales-repo-path', 'build/compare-locales', '--use-pymake', '--generate-partials', '--tooltool-manifest', 'mail/config/tooltool-manifests/win32/releng.manifest', '--tooltool-script', 'python', '--tooltool-script', '/c/mozilla-build/tooltool.py', '--tooltool-url', 'https://api.pub.build.mozilla.org/tooltool/', '--balrog-api-root', 'https://aus4-admin.mozilla.org/api', '--balrog-username', 'tbirdbld', '--credentials-file', 'oauth.txt', '--bucket-prefix', 'net-mozaws-prod-delivery'], attempting to kill
SIGKILL failed to kill process

**Full details are available at:
 http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-comm-beta-win32_repack_7%2F10/builds/0
I'm not sure what's going on here. Might be worth asking Axel...looks like something is going wrong in the l10n build system.
(In reply to Ben Hearsum (:bhearsum) from comment #1)
> I'm not sure what's going on here. Might be worth asking Axel...looks like
> something is going wrong in the l10n build system.
Flags: needinfo?(l10n)
No idea, and I don't have access to the bb masters, so no access to the logs.

Is this the first beta in the 43 cycle for tb? Also, why would this be a 43 instead of a 44?
Flags: needinfo?(l10n)
(In reply to Axel Hecht [:Pike] from comment #3)
> No idea, and I don't have access to the bb masters, so no access to the logs.

Public links are under https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/

Mac https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_4-bm86-build1-build5.txt.gz

Windows  https://archive.mozilla.org/pub/thunderbird/candidates/43.0b1-candidates/build3/logs/release-comm-beta-win32_repack_7-bm72-build1-build0.txt.gz
 

> Is this the first beta in the 43 cycle for tb? 

It is the first beta aka b1, but the third build attempt because we've had numerous problems - automation, etc.


> Also, why would this be a 43 instead of a 44?

Because of the above, and our desire to use 44 beta for part 2 of the sha-1 fixes
Flags: needinfo?(l10n)
Given it's lproj, maybe something to match https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc is missing? With those, I'm really stumbling in the dark, maybe glandium has a better idea.
Flags: needinfo?(l10n)
(In reply to Axel Hecht [:Pike] from comment #5)
> Given it's lproj, maybe something to match
> https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc is missing?

That was done in bug 1185268, so this should be in current comm-beta. But if it were that, it would fail for all locales, presumably.

The thing to note, however, is that it works apparently once, and fails in subsequent runs, which, considering the failing command is a `mv`, can very well mean that there's something *not* happening in between the different locale runs that should be happening (because even before https://hg.mozilla.org/mozilla-central/rev/a1587270e7fc , it was a `mv` as well).
Oh, those might actually be red herrings. The first real error seems to be:

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/comm-beta/mozilla/obj-l10n'

Yay, one more /mozilla/ path issue.
(In reply to Mike Hommey [:glandium] from comment #7)
> Oh, those might actually be red herrings. The first real error seems to be:
> 
> OSError: [Errno 2] No such file or directory:
> '/builds/slave/tb-rel-c-beta-m64_rpk_4-000000/comm-beta/mozilla/obj-l10n'
> 
> Yay, one more /mozilla/ path issue.

Does this help nail it?
Flags: needinfo?(rail)
Flags: needinfo?(aleth)
Something tells me it may be related to unset MOZ_CURRENT_PROJECT...

https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#184 tries to compare paths set in mozinfo.json and fails, because it skips https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#180
Flags: needinfo?(rail)
What can possibly go wrong?!
Attachment #8702974 - Flags: review?(bugspam.Callek)
I pushed it
remote:   https://hg.mozilla.org/build/tools/rev/7d6cf1fdcf03
remote:   https://hg.mozilla.org/build/tools/rev/2c636f6a0ec6

rerunning one of the repacks to see if it worked
Comment on attachment 8702974 [details] [diff] [review]
tb_mac_repacks.diff

if it works sure
Attachment #8702974 - Flags: review?(bugspam.Callek) → review+
The trick above didn't work, so I played a bit on a builder and finally added a dirty hack to work around the /mozilla/ path issue: http://hg.mozilla.org/build/tools/file/a72c9646a5bd/scripts/l10n/create-release-repacks.py#l116

This is just a work around and should be addressed in a better way.
More fun:

command: START
command: c:\mozilla-build\python27\python.exe c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_7-000000\comm-beta/build/pymake/make.py --no-print-directory echo-variable-CHECKSUM_FILE AB_CD=pt-PT
command: cwd: c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_7-000000\comm-beta\obj-l10n\mail\locales
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'pt-PT', 'MOZ_SIGN_CMD': 'python c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\signtool.py --cachedir c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\signing_cache -t c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\token -n c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\nonce -c c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'WIN32_INSTALLER_IN': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\thunderbird.exe', 'UPLOAD_SSH_KEY': '~\\.ssh\\tbirdbld_dsa', 'LOCALE_MERGEDIR': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\comm-beta\\\\\\\\obj-l10n\\\\\\\\mail\\\\\\\\locales\\\\\\\\merged', 'MOZILLA_OFFICIAL': '1', 'COMM_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/comm-beta/obj-l10n/dist/host/bin/mar.exe', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'ZIP_IN': 'c:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\tb-rel-c-beta-w32_rpk_7-000000\\\\\\\\thunderbird.zip', 'UPLOAD_USER': 'tbirdbld', 'USE_PYMAKE': '1', 'MBSDIFF': '/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/comm-beta/obj-l10n/dist/host/bin/mbsdiff.exe', 'UPLOAD_EXTRA_FILES': 'update/win32/pt-PT/thunderbird-42.0b1-43.0b1.partial.mar update/win32/pt-PT/thunderbird-42.0b1-43.0b1.partial.mar.asc update/win32/pt-PT/thunderbird-42.0b2-43.0b1.partial.mar update/win32/pt-PT/thunderbird-42.0b2-43.0b1.partial.mar.asc update/win32/pt-PT/thunderbird-41.0b2-43.0b1.partial.mar update/win32/pt-PT/thunderbird-41.0b2-43.0b1.partial.mar.asc', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 3 -v 43.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_43_0b1_RELEASE', 'MOZ_PKG_VERSION': '43.0b1', 'MBSDIFF_HOOK': 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'}
command: END (0.25s elapsed)

command: output:
'../../dist/win32/pt-PT//thunderbird-43.0b1.checksums'


The following tracebacks were detected during repacks:
nl:
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_7-000000/scripts/scripts/l10n/create-release-repacks.py", line 146, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 22] invalid mode ('r') or filename: "c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_7-000000\\comm-beta\\obj-l10n\\dist\\win32\\nl\\thunderbird-43.0b1.checksums'\r"


Sounds like something has changed (on the builders?) and the command above uses CRLF.


http://hg.mozilla.org/build/tools/rev/de42cac7de69 should fix it.
There were a few more
Thunderbird 43.0b1 build3: failed at repack_6/10 on win32  11:16pm EST
which succeeded two hours later
Thunderbird 43.0b1 build3: completed repack_6/10 on win32  1:19am EST

Then several of the type
Thunderbird 43.0b1 build3: failed at update_verify_beta_2/6 on linux
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #15)
> There were a few more
> Thunderbird 43.0b1 build3: failed at repack_6/10 on win32  11:16pm EST
> which succeeded two hours later
> Thunderbird 43.0b1 build3: completed repack_6/10 on win32  1:19am EST
> 
> Then several of the type
> Thunderbird 43.0b1 build3: failed at update_verify_beta_2/6 on linux

Those a different, bug 1209376
Should this be resolved since the 43 beta shipped?
Flags: needinfo?(aleth)
(In reply to aleth [:aleth] from comment #17)
> Should this be resolved since the 43 beta shipped?

depends on how rail wants to proceed, considering ...

(In reply to Rail Aliiev [:rail] from comment #13)
> The trick above didn't work, so I played a bit on a builder and finally
> added a dirty hack to work around the /mozilla/ path issue:
> http://hg.mozilla.org/build/tools/file/a72c9646a5bd/scripts/l10n/create-
> release-repacks.py#l116
> 
> This is just a work around and should be addressed in a better way.
Assignee: nobody → rail
Flags: needinfo?(rail)
It works now, but the work around is very hacky. We may be hit again some time in the future again, unless we fix the root clause, which is somewhere in the build system AFAIK.
Assignee: rail → nobody
Flags: needinfo?(rail)
Resolving for now.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 46.0
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: