Closed Bug 951238 Opened 6 years ago Closed 6 years ago

Android builders and Android l10n repack machines have different aapt versions

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox27- affected, firefox28 affected, firefox29 affected)

RESOLVED FIXED
Tracking Status
firefox27 - affected
firefox28 --- affected
firefox29 --- affected

People

(Reporter: nalexander, Assigned: aki)

References

Details

Attachments

(3 files, 3 obsolete files)

This manifests itself in the current results of Bug 923950.

I deduce the title because of the output about the missing '--output-text-symbols' option, although I haven't verified that the versions are different.  This is bad, both because of the errors this caused in Bug 923950, but more because l10n repacks use some properties of multiple runs of aapt that are true now, and likely to be true in the future, but are not explicitly guaranteed.  So we shouldn't tempt fate by using multiple versions if we can avoid it.

The relevant line that succeeds on a builder is [1] and that fails on an l10n slave is [2]:

[1]: /builds/slave/m-cen-and-00000000000000000000/build/android-sdk-linux/platforms/android-16/../..//platform-tools/aapt package -f -M AndroidManifest.xml -I /builds/slave/m-cen-and-00000000000000000000/build/android-sdk-linux/platforms/android-16/android.jar --auto-add-overlay -S /builds/slave/m-cen-and-00000000000000000000/build/mobile/android/base/crashreporter/res -S /builds/slave/m-cen-and-00000000000000000000/build/mobile/android/base/resources -S /builds/slave/m-cen-and-00000000000000000000/build/mobile/android/branding/nightly/res -S /builds/slave/m-cen-and-00000000000000000000/build/obj-firefox/mobile/android/base/res --custom-package org.mozilla.gecko --non-constant-id -F gecko-nodeps.ap_ -J gecko-nodeps/ --output-text-symbols gecko-nodeps/

[2]: 04:11:01     INFO -  /tools/android-sdk-r16/platforms/android-16/../..//platform-tools/aapt package -f -M AndroidManifest.xml -I /tools/android-sdk-r16/platforms/android-16/android.jar --auto-add-overlay -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/crashreporter/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/resources -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/branding/nightly/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/res --custom-package org.mozilla.gecko --non-constant-id -F gecko-nodeps.ap_ -J gecko-nodeps/ --output-text-symbols gecko-nodeps/
Blocks: 923950
Found in triage.
Component: Release Automation → General Automation
QA Contact: bhearsum → catlee
rail, do you have time to look at this?  It is technically possible to work around it in the tree, but revving the Android platform-tools on the l10n slaves is a better long-term solution.  Given that we want to uplift this ASAP and we're still not sure if Bug 923950 is fixed we need to make a call.
Flags: needinfo?(rail)
Looks like the l10n repacks script doesn't use tooltool to fetch proper android tools specified in mobile/android/config/tooltool-manifests/android/releng.manifest.

aki, do you have any idea how trivial it'd be to make the script switch to tooltool based android sdk/ndk?
Flags: needinfo?(rail) → needinfo?(aki)
Looks like it's not difficult, the TooltoolMixin gives a lot of what we need.
http://hg.mozilla.org/build/mozharness/file/343abbdea121/mozharness/mozilla/tooltool.py#l15

It doesn't currently do the bootstrap section, but that doesn't look hard to add:
http://hg.mozilla.org/build/tools/file/131f4edf8e0c/scripts/tooltool/fetch_and_unpack.sh#l24

Then it looks like the mozconfig will need updating:

04:07:54     INFO -  if test `uname -m` = 'x86_64'; then
04:07:54     INFO -    ac_add_options --with-android-ndk="$topsrcdir/android-ndk"
04:07:54     INFO -    ac_add_options --with-android-sdk="$topsrcdir/android-sdk-linux/platforms/android-16"
04:07:54     INFO -    ac_add_options --with-android-gnu-compiler-version=4.7
04:07:54     INFO -  else
04:07:54     INFO -    ac_add_options --with-android-ndk="/tools/android-ndk-r8c"
04:07:54     INFO -    ac_add_options --with-android-sdk="/tools/android-sdk-r16/platforms/android-16"
04:07:54     INFO -    ac_add_options --with-android-gnu-compiler-version=4.6
04:07:54     INFO -  fi

04:07:56     INFO -  Adding configure options from /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/.mozconfig:
04:07:56     INFO -    --enable-crashreporter
04:07:56     INFO -    --enable-release
04:07:56     INFO -    --enable-elf-hack
04:07:56     INFO -    --enable-application=mobile/android
04:07:56     INFO -    --target=arm-linux-androideabi
04:07:56     INFO -    --with-android-ndk=/tools/android-ndk-r8c
04:07:56     INFO -    --with-android-sdk=/tools/android-sdk-r16/platforms/android-16
Flags: needinfo?(aki)
Assignee: nobody → aki
Attached patch silence_pyflakesSplinter Review
Not blocking this bug, but I got sick of seeing these warnings in unit.sh.

The b2g_desktop_unittest.py changes are the significant ones, and might be worth testing on Cypress or Ash; the android_emulator_build.py changes are cosmetic.
Attachment #8349806 - Flags: review?(ahalberstadt)
Attached patch [needs testing] mozconfig.diff (obsolete) — Splinter Review
Afaik we don't use the x86 or armv6 l10n mozconfigs, but edited anyway for consistency.

I'll try testing in staging tomorrow.
Update the rest of the config files while I'm at it.
Attachment #8349807 - Attachment is obsolete: true
Attachment #8349806 - Flags: review?(ahalberstadt) → review+
Kicked off a staging nightly; I forget if that triggers l10n nightlies or if I need to do that myself.

I'm planning on also running a staging release repack.  We could potentially skip that if we think the patches will fix it, since they're already broken.  Probably depends on how long the testing takes.
Good news: the tooltool extraction works, after I fixed the patches quite a bit.

Bad news: the tooltool package for the build is 64bit; the repack is 32bit.  Trying to use the tooltool package for 32 bit doesn't work.  Trying to switch the repack to 64bit hasn't been working so far.  The way forward may be creating a new 32bit tooltool package, unless we think it's worth switching the repack to 64bit.
Making progress on 64 bit.

... Is there a way to not look for libz.so.1 and look for libz.so instead?
(guessing no, I have to softlink =\ )

10:05:14  WARNING -  	   make -C /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base gecko-nodeps.ap_ && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/gecko-nodeps.ap_ /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && ( (test ! -f /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/R.txt && echo "*** Warning: The R.txt that is being packaged might not agree with the R.txt that was built. This is normal during l10n repacks.") || diff /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/R.txt /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/gecko-nodeps/R.txt >/dev/null || (echo "*** Error: The R.txt that was built and the R.txt that is being packaged are not the same. Rebuild mobile/android/base and re-package." && exit 1)) && ( cd fennec && mkdir -p lib/armeabi-v7a && mv libmozglue.so lib/libplugin-container.so lib/armeabi-v7a && unzip -o /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && rm /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && /usr/bin/zip -0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so && /usr/bin/zip -r9D /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ resources.arsc AndroidManifest.xml chrome components defaults modules hyphenation res lib extensions application.ini package-name.txt platform.ini greprefs.js browserconfig.properties blocklist.xml chrome.manifest update.locale removed-files recommended-addons.json distribution -x classes.dex  libmozglue.so lib/libplugin-container.so assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so &&  mkdir -p assets/ && mv omni.ja assets/ &&  /usr/bin/zip -0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ assets/omni.ja) && rm -f /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && /usr/bin/zip -j0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk fennec/classes.dex && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko-unsigned-unaligned.apk && python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H signing4.srv.releng.scl3.mozilla.com:9110 -f jar /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/tools/zipalign -f -v 4 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk fennec-29.0a1.be.android-arm.apk && echo 'Testing is disabled - No Android Robocop for you' && echo 'Testing is disabled - No Android Background tests for you' && echo 'GeckoView library packaging is disabled'
10:05:14     INFO -  make[2]: Entering directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base'
10:05:14     INFO -  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platform-tools/aapt package -f -M AndroidManifest.xml -I /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platforms/android-16/android.jar --auto-add-overlay -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/crashreporter/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/resources -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/branding/nightly/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/res --custom-package org.mozilla.gecko --non-constant-id -F gecko-nodeps.ap_ -J gecko-nodeps/ --output-text-symbols gecko-nodeps/
10:05:14     INFO -  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platform-tools/aapt: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
10:05:14    ERROR -  make[2]: *** [.aapt.nodeps] Error 127
10:05:14     INFO -  make[2]: Leaving directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base'
10:05:14    ERROR -  make[1]: *** [repackage-zip] Error 2
10:05:14     INFO -  make[1]: Leaving directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/locales'
10:05:14     INFO -  make: *** [repackage-zip-be] Error 2
10:05:14    ERROR - Return code: 2
(In reply to Aki Sasaki [:aki] from comment #13)
> Making progress on 64 bit.
> 
> ... Is there a way to not look for libz.so.1 and look for libz.so instead?
> (guessing no, I have to softlink =\ )

aki, thanks for your work on this.  Doesn't this mean that the libz configuration is different between builders and repackers, too?  I worry that this rabbit hole is deep.

I can work around using this option in code.  It's ugly to add a work-around, and it takes a nightly cycle to test, but before we dig this rabbit out let's estimate the depth of your hole.
Possibly... right now the repacks are running in linux32 land.

I found a libz.so.1 in /lib64, so I'm adding that to LD_LIBRARY_PATH to see if it fixes anything.
Hm, I seem to be mistaken.  But builds don't need zlib at all aiui, so that may be why they're not hitting this.

I wonder if I can explicitly install zlib 32bit?

10:50:41  WARNING -  	   make -C /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base gecko-nodeps.ap_ && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/gecko-nodeps.ap_ /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && ( (test ! -f /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/R.txt && echo "*** Warning: The R.txt that is being packaged might not agree with the R.txt that was built. This is normal during l10n repacks.") || diff /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/R.txt /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/gecko-nodeps/R.txt >/dev/null || (echo "*** Error: The R.txt that was built and the R.txt that is being packaged are not the same. Rebuild mobile/android/base and re-package." && exit 1)) && ( cd fennec && mkdir -p lib/armeabi-v7a && mv libmozglue.so lib/libplugin-container.so lib/armeabi-v7a && unzip -o /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && rm /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ && /usr/bin/zip -0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so && /usr/bin/zip -r9D /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ resources.arsc AndroidManifest.xml chrome components defaults modules hyphenation res lib extensions application.ini package-name.txt platform.ini greprefs.js browserconfig.properties blocklist.xml chrome.manifest update.locale removed-files recommended-addons.json distribution -x classes.dex  libmozglue.so lib/libplugin-container.so assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so assets/libmozalloc.so assets/libnss3.so assets/libxul.so assets/libnssckbi.so assets/libfreebl3.so assets/libsoftokn3.so assets/libomxplugin.so assets/libomxplugingb.so assets/libomxplugingb235.so assets/libomxpluginhc.so assets/libomxpluginfroyo.so &&  mkdir -p assets/ && mv omni.ja assets/ &&  /usr/bin/zip -0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ assets/omni.ja) && rm -f /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.ap_ /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && /usr/bin/zip -j0 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk fennec/classes.dex && cp /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko-unsigned-unaligned.apk && python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H signing4.srv.releng.scl3.mozilla.com:9110 -f jar /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk && /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/tools/zipalign -f -v 4 /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/gecko.apk fennec-29.0a1.ar.android-arm.apk && echo 'Testing is disabled - No Android Robocop for you' && echo 'Testing is disabled - No Android Background tests for you' && echo 'GeckoView library packaging is disabled'
10:50:41     INFO -  make[2]: Entering directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base'
10:50:41     INFO -  mkdir -p 'gecko-nodeps/'
10:50:41     INFO -  rm -f '/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/AndroidManifest.xml'
10:50:41     INFO -  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/_virtualenv/bin/python -m mozbuild.action.preprocessor --depend .deps/AndroidManifest.xml.pp  -DANDROID_VERSION_CODE=2014010203 -DMOZ_ANDROID_SHARED_ID="org.mozilla.fennec.sharedID" -DMOZ_ANDROID_SHARED_ACCOUNT_TYPE="org.mozilla.fennec_sync" -DMOZ_ANDROID_SHARED_FXACCOUNT_TYPE="org.mozilla.fennec_account" -DMOZ_APP_BUILDID=20140102030203 -DMOZ_BUILD_TIMESTAMP=1388688641 -DUA_BUILDID=20140102  -DMOZ_ANDROID_ANR_REPORTER=1 -DMOZ_UPDATER='1' -DANDROID_PACKAGE_NAME='org.mozilla.fennec' -DANDROID_CPU_ARCH='armeabi-v7a' -DCPU_ARCH='arm' -DGRE_MILESTONE='29.0a1' -DMOZILLA_OFFICIAL='1' -DMOZ_APP_BASENAME='Fennec' -DMOZ_APP_DISPLAYNAME='Nightly' -DMOZ_APP_ID='{aa3c5121-dab2-40e2-81ca-7ea25febc110}' -DMOZ_APP_NAME='fennec' -DMOZ_APP_VENDOR='Mozilla' -DMOZ_APP_VERSION='29.0a1' -DMOZ_CHILD_PROCESS_NAME='lib/libplugin-container.so' -DMOZ_CRASHREPORTER='F' -DMOZ_UPDATE_CHANNEL='nightly' -DOMNIJAR_NAME='assets/omni.ja' -DOS_TARGET='Android' -DTARGET_XPCOM_ABI='arm-eabi-gcc3' -DMANGLED_ANDROID_PACKAGE_NAME='org.mozilla.fennec' -DMOZ_APP_ABI='arm-eabi-gcc3' -DMOZ_MIN_CPU_VERSION=7 -DNO_NSPR_10_SUPPORT -DANDROID='1' -DCROSS_COMPILE='1' -DMOZ_THUMB2='1' -DHAVE_ARM_SIMD='1' -DHAVE_ARM_NEON='1' -DBUILD_ARM_NEON='1' -DJSGC_INCREMENTAL='1' -DMOZ_ENABLE_PROFILER_SPS='1' -DMOZILLA_VERSION='"29.0a1"' -DMOZILLA_VERSION_U='29.0a1' -DMOZILLA_UAVERSION='"29.0"' -DNO_PW_GECOS='1' -DXP_LINUX='1' -DMOZ_LINKER='1' -DD_INO='d_ino' -DMOZ_DEBUG_SYMBOLS='1' -DSTDC_HEADERS='1' -DHAVE_SSIZE_T='1' -DHAVE_ST_BLKSIZE='1' -DHAVE_SIGINFO_T='1' -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE='1' -DHAVE_VISIBILITY_ATTRIBUTE='1' -DHAVE_DIRENT_H='1' -DHAVE_GETOPT_H='1' -DHAVE_MEMORY_H='1' -DHAVE_UNISTD_H='1' -DHAVE_MALLOC_H='1' -DHAVE_SYS_SYSMACROS_H='1' -DHAVE_LINUX_QUOTA_H='1' -DHAVE_LINUX_RTNETLINK_H='1' -DHAVE_SYS_TYPES_H='1' -DHAVE_NETINET_IN_H='1' -DHAVE_BYTESWAP_H='1' -DHAVE_SYS_CDEFS_H='1' -DHAVE_DLOPEN='1' -DHAVE_DLADDR='1' -DHAVE_MEMMEM='1' -DNO_X11='1' -DHAVE_LOCALTIME_R='1' -DHAVE_CLOCK_MONOTONIC='1' -DVA_COPY='va_copy' -DHAVE_VA_COPY='1' -DMALLOC_H='<malloc.h>' -DHAVE_STRNDUP='1' -DHAVE_MEMALIGN='1' -DHAVE_VALLOC='1' -DHAVE_I18N_LC_MESSAGES='1' -DNS_ATTR_MALLOC='__attribute__((malloc))' -DNS_WARN_UNUSED_RESULT='__attribute__((warn_unused_result))' -DNIGHTLY_BUILD='1' -DMOZ_UPDATE_CHANNEL='nightly' -DEARLY_BETA_OR_EARLIER='1' -DMOZ_BUILD_APP='mobile/android' -DMOZ_WIDGET_ANDROID='1' -DMOZ_PDF_PRINTING='1' -DMOZ_INSTRUMENT_EVENT_LOOP='1' -DUSE_ARM_KUSER='1' -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_ANDROID_HISTORY='1' -DMOZ_ANDROID_OMTC='1' -DMOZ_ANDROID_BEAM='1' -DIBMBIDI='1' -DACCESSIBILITY='1' -DNS_PRINTING='1' -DNS_PRINT_PREVIEW='1' -DMOZ_WEBRTC='1' -DMOZ_WEBRTC_ASSERT_ALWAYS='1' -DMOZ_WEBRTC_SIGNALING='1' -DMOZ_PEERCONNECTION='1' -DMOZ_SCTP='1' -DMOZ_SRTP='1' -DMOZ_SAMPLE_TYPE_S16='1' -DMOZ_WEBSPEECH='1' -DMOZ_RAW='1' -DMOZ_OGG='1' -DATTRIBUTE_ALIGNED_MAX='64' -DMOZ_WEBM='1' -DMOZ_MEDIA_PLUGINS='1' -DMOZ_MEDIA_NAVIGATOR='1' -DMOZ_OMX_PLUGIN='1' -DMOZ_VPX='1' -DMOZ_VPX_ERROR_CONCEALMENT='1' -DVPX_ARM_ASM='1' -DMOZ_WAVE='1' -DMOZ_TREMOR='1' -DMOZ_OPUS='1' -DENABLE_SYSTEM_EXTENSION_DIRS='1' -DMOZ_WEBGL='1' -DMOZ_CRASHREPORTER='1' -DMOZ_CRASHREPORTER_ENABLE_PERCENT='100' -DLIBJPEG_TURBO_ARM_ASM='1' -DMOZ_USE_NATIVE_POPUP_WINDOWS='1' -DMOZ_TREE_FREETYPE='1' -DHAVE_FT_BITMAP_SIZE_Y_PPEM='1' -DHAVE_FT_GLYPHSLOT_EMBOLDEN='1' -DHAVE_FT_LOAD_SFNT_TABLE='1' -DMOZ_SIGNING='1' -DMOZ_UPDATER='1' -DMOZ_DISABLE_CRYPTOLEGACY='1' -DMOZ_FEEDS='1' -DMOZ_SAFE_BROWSING='1' -DMOZ_URL_CLASSIFIER='1' -DGL_PROVIDER_='1' -DMOZ_GFX_OPTIMIZE_MOBILE='1' -DMOZ_LOGGING='1' -DMOZ_MEMORY='1' -DMOZ_MEMORY_LINUX='1' -DMOZ_MEMORY_ANDROID='1' -DJSGC_INCREMENTAL='1' -DJS_CRASH_DIAGNOSTICS='1' -DMOZ_B2G_CERTDATA='1' -DMOZ_PAY='1' -DHAVE__UNWIND_BACKTRACE='1' -DJS_DEFAULT_JITREPORT_GRANULARITY='3' -DMOZ_OMNIJAR='1' -DMOZ_USER_DIR='".mozilla"' -DMOZ_STATIC_JS='1' -DMOZ_TREE_PIXMAN='1' -DHAVE_STDINT_H='1' -DHAVE_INTTYPES_H='1' -DMOZ_TREE_CAIRO='1' -DHAVE_UINT64_T='1' -DMOZ_ENABLE_SKIA='1' -DUSE_SKIA='1' -DSK_BUILD_FOR_ANDROID_NDK='1' -DUSE_SKIA_GPU='1' -DMOZ_XUL='1' -DMOZ_PROFILELOCKING='1' -DBUILD_CTYPES='1' -DMOZ_SERVICES_HEALTHREPORT='1' -DMOZ_APP_COMPONENT_INCLUDE='"nsBrowserComponents.h"' -DMOZ_MACBUNDLE_ID='org.mozilla.nightly' -DMOZ_B2G_VERSION='"1.0.0"' -DMOZ_B2G_OS_NAME='""' -DMOZ_APP_UA_NAME='"Firefox"' -DMOZ_APP_UA_VERSION='"29.0a1"' -DFIREFOX_VERSION='29.0a1' -DMOZ_TELEMETRY_DISPLAY_REV='2' -DMOZ_DATA_REPORTING='1' -DMOZ_DLL_SUFFIX='".so"' -DXP_UNIX='1' -DNDEBUG -DTRIMMED '/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/AndroidManifest.xml.in' -o '/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/AndroidManifest.xml'
10:50:41     INFO -  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platform-tools/aapt package -f -M AndroidManifest.xml -I /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platforms/android-16/android.jar --auto-add-overlay -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/crashreporter/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/base/resources -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/mobile/android/branding/nightly/res -S /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base/res --custom-package org.mozilla.gecko --non-constant-id -F gecko-nodeps.ap_ -J gecko-nodeps/ --output-text-symbols gecko-nodeps/
10:50:41     INFO -  /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/android-sdk-linux/platform-tools/aapt: error while loading shared libraries: libz.so.1: wrong ELF class: ELFCLASS64
10:50:41    ERROR -  make[2]: *** [.aapt.nodeps] Error 127
10:50:41     INFO -  make[2]: Leaving directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/base'
10:50:41    ERROR -  make[1]: *** [repackage-zip] Error 2
10:50:41     INFO -  make[1]: Leaving directory `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/locales'
10:50:41     INFO -  make: *** [repackage-zip-ar] Error 2
This is the error I got when I ran on linux32:

22:30:20     INFO -  checking for android toolchain directory... configure: error: not found. Your --with-android-gnu-compiler-version may be wrong.

which was
22:30:17     INFO -    --with-android-gnu-compiler-version=4.7

And I see

./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/libexec/gcc
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/libexec/gcc/arm-linux-androideabi/4.7/plugin/gengtype
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/lib/gcc
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/arm-linux-androideabi/bin/g++
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/arm-linux-androideabi/bin/gcc

in the android-ndk dir, but no i686.
Ok, once I installed zlib i686, I hit a missing libstdc++.so.6 error.
Added the /tools/gcc-4.7.2-0moz1/lib* dirs to LD_LIBRARY_PATH and now the repacks work.

http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/rev/19a172ea3459#l1.13

Off to polish up the patches, test a beta repack, and feel icky about the hack.
Comment on attachment 8349810 [details] [diff] [review]
[needs testing] mozconfig.diff

Since we're now hitting the x86_64 block, no need to make any mozconfig changes.
Attachment #8349810 - Attachment is obsolete: true
Beta repack hits this:

16:28:45     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/obj-l10n/dist/l10n-stage/fennec/res/color/url_bar_title.xml'

Not sure if there's an issue with the beta locale repos or something in my staging setup or what.  I think moving forward is still a good idea, though.
Attached patch android-tooltoolSplinter Review
This patch:

* adds tooltool configs to all android single locale configs
* sets LD_LIBRARY_PATH per this bug :\
* moves android repacks to x86_64, and adds all the x86_64 mock packages
* removes the staging configs for the android nightly repacks (not sure they were used; diverging)
* allows for tuple commands in run_command() because of bug 706036
* adds bootstrap command support in tooltool
* adds tooltool support in mobile_l10n.py
Attachment #8349816 - Attachment is obsolete: true
Attachment #8355382 - Flags: review?(rail)
Attachment #8355382 - Flags: review?(rail) → review+
Comment on attachment 8355382 [details] [diff] [review]
android-tooltool

http://hg.mozilla.org/build/mozharness/rev/abfaf9d62533
Transplanted to production.
Rerunning a nightly first.
Attachment #8355382 - Flags: checked-in+
m-c 1/5 is green: https://tbpl.mozilla.org/php/getParsedLog.php?id=32524719&tree=Mozilla-Central

Respun the rest of m-c nightly repacks.

I'm not finding Aurora Android nightly repacks =\
I think I might spin the latest beta repacks to see what happens.
(In reply to Aki Sasaki [:aki] from comment #22)
> Comment on attachment 8355382 [details] [diff] [review]
> android-tooltool
> 
> http://hg.mozilla.org/build/mozharness/rev/abfaf9d62533
> Transplanted to production.
> Rerunning a nightly first.

Of course it was actually failed for other things, so backed out in:

  https://hg.mozilla.org/build/mozharness/rev/d42e3104edfd
  https://hg.mozilla.org/build/mozharness/rev/ff266e1efc39

Log snippet of failure is:

11:36:15     INFO - Running main action method: get_blobs
11:36:15     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x19e6c90>> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool/', 'fetch', '-m', u'/builds/slave/b2g_m-b26_12_emu_dep-000000000/build/gecko/b2g/config/emulator/releng-emulator.tt', '-o'],), kwargs: {'error_list': [{'substr': 'command not found', 'level': 'error'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1a775e0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1a6d8b0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': None}, attempt #1
11:36:15     INFO - Running command: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool/', 'fetch', '-m', u'/builds/slave/b2g_m-b26_12_emu_dep-000000000/build/gecko/b2g/config/emulator/releng-emulator.tt', '-o']
11:36:15     INFO - Copy/paste: /tools/tooltool.py --url http://runtime-binaries.pvt.build.mozilla.org/tooltool/ fetch -m /builds/slave/b2g_m-b26_12_emu_dep-000000000/build/gecko/b2g/config/emulator/releng-emulator.tt -o
11:36:16     INFO - Return code: 0
11:36:16     INFO - retry: Calling <bound method B2GBuild.run_command of <__main__.B2GBuild object at 0x19e6c90>> with args: ('/builds/slave/b2g_m-b26_12_emu_dep-000000000/build',), kwargs: {'error_list': [{'substr': 'command not found', 'level': 'error'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x1a775e0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1a6d8b0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': None}, attempt #1
11:36:16     INFO - Running command: /builds/slave/b2g_m-b26_12_emu_dep-000000000/build
11:36:16     INFO -  /bin/sh: /builds/slave/b2g_m-b26_12_emu_dep-000000000/build: is a directory
11:36:16    ERROR - Return code: 126


Which is probably due to param ordering issues as it relates to http://hg.mozilla.org/build/mozharness/diff/abfaf9d62533/mozharness/mozilla/tooltool.py
Attachment #8355382 - Flags: checked-in+ → checked-in-
Attached patch fix_b2gSplinter Review
This is the only tooltool_fetch() call that doesn't explicitly specify output_dir=
Attachment #8355632 - Flags: review?(bugspam.Callek)
Attachment #8355632 - Flags: review?(bugspam.Callek) → review+
I did find the Aurora repacks, though they don't show up on tbpl.
Those are still broken:

12:01:37    ERROR -  Traceback (most recent call last):
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/toolkit/mozapps/installer/l10n-repack.py", line 48, in <module>
12:01:37     INFO -      main()
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/toolkit/mozapps/installer/l10n-repack.py", line 44, in main
12:01:37     INFO -      l10n.repack(args.build, args.l10n, args.non_resource, NON_CHROME)
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/python/mozbuild/mozpack/packager/l10n.py", line 173, in repack
12:01:37     INFO -      copier.copy(source, skip_if_older=False)
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/python/mozbuild/mozpack/copier.py", line 293, in copy
12:01:37     INFO -      if f.copy(destfile, skip_if_older):
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/python/mozbuild/mozpack/copier.py", line 438, in copy
12:01:37     INFO -      file.copy(deflater, skip_if_older)
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/python/mozbuild/mozpack/files.py", line 105, in copy
12:01:37     INFO -      shutil.copyfileobj(self.open(), dest)
12:01:37     INFO -    File "/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/python/mozbuild/mozpack/files.py", line 133, in open
12:01:37     INFO -      return open(self.path, 'rb')
12:01:37     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/obj-l10n/dist/l10n-stage/fennec/res/drawable-xxhdpi/icon.png'
12:01:37    ERROR -  make[1]: *** [repackage-zip] Error 1
12:01:37     INFO -  make[1]: Leaving directory `/builds/slave/rel-m-beta-and_rpk_1-000000000/build/mozilla-beta/obj-l10n/mobile/android/locales'
12:01:37     INFO -  make: *** [repackage-zip-cs] Error 2
12:01:37    ERROR - Return code: 2
Oops, wrong traceback.  This is Aurora:

11:58:52    ERROR -  Traceback (most recent call last):
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/toolkit/mozapps/installer/l10n-repack.py", line 48, in <module>
11:58:52     INFO -      main()
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/toolkit/mozapps/installer/l10n-repack.py", line 44, in main
11:58:52     INFO -      l10n.repack(args.build, args.l10n, args.non_resource, NON_CHROME)
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/packager/l10n.py", line 173, in repack
11:58:52     INFO -      copier.copy(source, skip_if_older=False)
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/copier.py", line 293, in copy
11:58:52     INFO -      if f.copy(destfile, skip_if_older):
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/copier.py", line 438, in copy
11:58:52     INFO -      file.copy(deflater, skip_if_older)
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/files.py", line 105, in copy
11:58:52     INFO -      shutil.copyfileobj(self.open(), dest)
11:58:52     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/files.py", line 133, in open
11:58:52     INFO -      return open(self.path, 'rb')
11:58:52     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/dist/l10n-stage/fennec/res/drawable-large-xhdpi-v11/ic_menu_forward.png'
11:58:52    ERROR -  make[1]: *** [repackage-zip] Error 1
11:58:52     INFO -  make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales'
11:58:52     INFO -  make: *** [repackage-zip-et] Error 2
11:58:52    ERROR - Return code: 2
B2g builds ran fine on Cypress.  I'll wait for the scheduled merge next week, though.
Beta's hitting this:

15:39:29    ERROR -  Traceback (most recent call last):
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/toolkit/mozapps/installer/l10n-repack.py", line 48, in <module>
15:39:29     INFO -      main()
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/toolkit/mozapps/installer/l10n-repack.py", line 44, in main
15:39:29     INFO -      l10n.repack(args.build, args.l10n, args.non_resource, NON_CHROME)
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/python/mozbuild/mozpack/packager/l10n.py", line 173, in repack
15:39:29     INFO -      copier.copy(source, skip_if_older=False)
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/python/mozbuild/mozpack/copier.py", line 293, in copy
15:39:29     INFO -      if f.copy(destfile, skip_if_older):
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/python/mozbuild/mozpack/copier.py", line 438, in copy
15:39:29     INFO -      file.copy(deflater, skip_if_older)
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/python/mozbuild/mozpack/files.py", line 105, in copy
15:39:29     INFO -      shutil.copyfileobj(self.open(), dest)
15:39:29     INFO -    File "/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/python/mozbuild/mozpack/files.py", line 133, in open
15:39:29     INFO -      return open(self.path, 'rb')
15:39:29     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/obj-l10n/dist/l10n-stage/fennec/res/drawable-xlarge-xhdpi-v11/ic_menu_bookmark_remove.png'
15:39:29    ERROR -  make[1]: *** [repackage-zip] Error 1
15:39:29     INFO -  make[1]: Leaving directory `/builds/slave/rel-m-beta-and_rpk_3-000000000/build/mozilla-beta/obj-l10n/mobile/android/locales'
15:39:29     INFO -  make: *** [repackage-zip-es-AR] Error 2
15:39:29    ERROR - Return code: 2
Aurora's getting the below.
We're using tooltool and should have the same aapt as builds now.
:nalexander, do you want me to resolve this bug, morph it and reassign, or ?

   ERROR -  Traceback (most recent call last):
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/toolkit/mozapps/installer/l10n-repack.py", line 48, in <module>
13:05:09     INFO -      main()
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/toolkit/mozapps/installer/l10n-repack.py", line 44, in main
13:05:09     INFO -      l10n.repack(args.build, args.l10n, args.non_resource, NON_CHROME)
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/packager/l10n.py", line 173, in repack
13:05:09     INFO -      copier.copy(source, skip_if_older=False)
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/copier.py", line 293, in copy
13:05:09     INFO -      if f.copy(destfile, skip_if_older):
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/copier.py", line 438, in copy
13:05:09     INFO -      file.copy(deflater, skip_if_older)
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/files.py", line 105, in copy
13:05:09     INFO -      shutil.copyfileobj(self.open(), dest)
13:05:09     INFO -    File "/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/python/mozbuild/mozpack/files.py", line 133, in open
13:05:09     INFO -      return open(self.path, 'rb')
13:05:09     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/obj-l10n/dist/l10n-stage/fennec/res/drawable-xxhdpi/icon.png'
13:05:10    ERROR -  make[1]: *** [repackage-zip] Error 1
13:05:10     INFO -  make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_2-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales'
13:05:10     INFO -  make: *** [repackage-zip-he] Error 2
Flags: needinfo?(nalexander)
Crap, now m-c repacks are getting this:

13:00:48     INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '-q', '--cwd', '/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/locales', '--unpriv', '--shell', '/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/tools/jdk6/bin:/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing4.srv.releng.scl3.mozilla.com:9100,signing5.srv.releng.scl3.mozilla.com:9100,signing6.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ JAVA_HOME=/tools/jdk6 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US CCACHE_DIR=/builds/ccache SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 "MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H signing4.srv.releng.scl3.mozilla.com:9100 -H signing5.srv.releng.scl3.mozilla.com:9100 -H signing6.srv.releng.scl3.mozilla.com:9100" LC_ALL=C _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=bld-centos6-hp-018.build.scl1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make installers-ar'] in /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/locales
13:00:48     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --cwd /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/mobile/android/locales --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/tools/jdk6/bin:/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing4.srv.releng.scl3.mozilla.com:9100,signing5.srv.releng.scl3.mozilla.com:9100,signing6.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ JAVA_HOME=/tools/jdk6 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US CCACHE_DIR=/builds/ccache SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H signing4.srv.releng.scl3.mozilla.com:9100 -H signing5.srv.releng.scl3.mozilla.com:9100 -H signing6.srv.releng.scl3.mozilla.com:9100\" LC_ALL=C _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=bld-centos6-hp-018.build.scl1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make installers-ar"
13:00:48     INFO -  rm -f -rf /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/l10n-stage/fennec
13:00:48     INFO -  rm -f ../../../mobile/android/base/res/values-*/strings.xml
13:00:48    ERROR -  make: *** No rule to make target `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-29.0a1.en-US.android-arm.apk', needed by `/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/l10n-stage/fennec'.  Stop.
13:00:48    ERROR - Return code: 2
I'll just mark RESO FIXED since the summary request is resolved.
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(nalexander)
Resolution: --- → FIXED
Merged mozharness (not getting CCed to this bug).
Not tracking for release tracking per latest comment in 923950
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.