1057025 - Android l10n nightly-3 on mozilla-aurora branch is failing on "make lg failed in make installers-lg!"

Status: RESOLVED DUPLICATE of bug 1062880

(Mozilla Localizations :: lg / Luganda, defect)

Description

[Kim Moir [:kmoir] ET]

Noticed this while working on bug 1019919.  

http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/Android%202.3%20mozilla-aurora%20l10n%20nightly-3/builds/2/steps/run_script/logs/stdio has more details.

Comment 1

[Francesco Lodolo [:flod]]

Would it be possible to report the relevant part of the log for folks without VPN access (like me)?

Comment 2

[Kim Moir [:kmoir] ET]

Here's the relevant part of the log, I'll attach the entire log too.

02:40:51     INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '-q', '--unpriv', '--shell', 'touch /builds/slave/m-aurora-and-l10n_3-0000000000/build/update/mozilla-aurora/Android_arm-eabi-gcc3/20140821004002/ko/partial.txt']
02:40:51     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --unpriv --shell "touch /builds/slave/m-aurora-and-l10n_3-0000000000/build/update/mozilla-aurora/Android_arm-eabi-gcc3/20140821004002/ko/partial.txt"
02:40:52     INFO - Return code: 0
02:40:52    ERROR - Can't determine the upload url for lg!
02:40:52    ERROR - You either need to run --upload-repacks before --create-nightly-snippets, or specify the 'snippet_base_url' in self.config!
02:40:52     INFO - Setting buildbot property lg_failure to lg failed in make installers-lg!  Can't create a snippet for lg without an upload url.
02:40:52     INFO - Writing buildbot properties ['lg_failure'] to /builds/slave/m-aurora-and-l10n_3-0000000000/properties/lg_failure
02:40:52     INFO - Writing to file /builds/slave/m-aurora-and-l10n_3-0000000000/properties/lg_failure

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1057025.log

entire log

Comment 4

[Axel Hecht [:Pike]]

It says that it's failing to determine an upload url, that looks like an automation problem.

My head can't parse the logs, so if there was something else, I won't find it.

Over to releng for investigation.

Comment 5

[Francesco Lodolo [:flod]]

I found

02:36:03    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base/res/values/strings.xml:1295: error: Error parsing XML: undefined entity

Referencing unknown entity `Omusubuzi` (vendorShortName known) at line 18, column 0 for crash_send_report_message3

http://hg.mozilla.org/releases/l10n/mozilla-aurora/lg/file/b9b667dabe65/mobile/android/base/android_strings.dtd#l19

<!ENTITY  crash_send_report_message3 "Gamba &Omusubuzi; ekikwata ku kkabenje asobole oku gitteleza">

Not sure if they're related.

Comment 6

[Axel Hecht [:Pike]]

Thanks for the investigation, yes, that's probably the case.

Back to the Lugandan component.

Please check the checks on pootle, http://mozilla.locamotion.org/lg/mobile/mobile/ complains just about variables 9 times, the other checks are good to go through, too.

CCing Kisaalu, could be that he's the editor on pootle, or at least connect us to that volunteer.

Comment 7

[Kim Moir [:kmoir] ET]

This patch fixed the issues in a nightly I retriggered and updated the balrog db, so I've retriggered the rest of the failing nightly l10n builds on m-a and m-c.

Comment 8

[Francesco Lodolo [:flod]]

@Kim
Which patch?

Comment 9

[Kim Moir [:kmoir] ET]

re: comment 6 .  Sorry I updated the wrong bug, this comment was supposed to be in bug 1019919.

Comment 10

[Kim Moir [:kmoir] ET]

As an aside all the android nightly single locale builds (on m-c and m-a) are failing with this error message and have been for weeks

07:57:29     INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64-android', '-q', '--cwd', '/builds/slave/m-cen-and-l10n_1-0000000000000/build', '--unpriv', '--shell', '/usr/bin/env MOZ_AUTOMATION=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing2.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing3.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing4.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ HG_SHARE_BASE_DIR=/builds/hg-shared MOZ_CRASHREPORTER_NO_REPORT=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 "MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100" LC_ALL=C TOOLTOOL_HOME=/builds _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=bld-linux64-spot-457.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 TOOLTOOL_CACHE=/builds/tooltool_cache tools/release/signing/verify-android-signature.sh --tools-dir=tools/ --nightly --apk=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk'] in /builds/slave/m-cen-and-l10n_1-0000000000000/build
07:57:29     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64-android -q --cwd /builds/slave/m-cen-and-l10n_1-0000000000000/build --unpriv --shell "/usr/bin/env MOZ_AUTOMATION=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing2.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing3.srv.releng.scl3.mozilla.com:9100,dmg:mac-signing4.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ HG_SHARE_BASE_DIR=/builds/hg-shared MOZ_CRASHREPORTER_NO_REPORT=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100\" LC_ALL=C TOOLTOOL_HOME=/builds _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=bld-linux64-spot-457.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 TOOLTOOL_CACHE=/builds/tooltool_cache tools/release/signing/verify-android-signature.sh --tools-dir=tools/ --nightly --apk=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk"
07:57:30     INFO -  TOOLS DIR: tools/
07:57:30     INFO -  Archive:  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk
07:57:30     INFO -    inflating: META-INF/NIGHTLY.DSA
07:57:30    FATAL -  Invalid
07:57:30    FATAL -  Signature is invalid!

Comment 11

[Jeff Beatty [:gueroJeff]]

Hi Kim,

Yes, they've been looking at this issue over in bug 1062880. It seems they still haven't come to a solution.

Comment 13

[Jordan Lund (:jlund)]

I am going to re-open this bug on the basis that the fix in 1062880 was verified and resolved yet chunk 3, specifically locale lg is still consistently failing.

example: https://treeherder.mozilla.org/logviewer.html#?job_id=567237&repo=mozilla-aurora

Comment 14

[Francesco Lodolo [:flod]]

Filing a new bug seems a smarter choice, given how much confusion there's already in this one (mixed l10n errors, build errors, etc.).