Closed Bug 714553 Opened 13 years ago Closed 13 years ago

Frequent Android XUL build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found"

Categories

(Firefox for Android Graveyard :: General, defect, P1)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
ARM
Android
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Unassigned)

References

Details

Attachments

(2 files, 2 obsolete files)

patch
1.01 KB, patch
khuey
: review-
ted
: review-
Details | Diff | Splinter Review
Avoid race condition for objdir/embedding/android/R.java dependencies
1.42 KB, patch
ted
: review+
Details | Diff | Splinter Review
e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=8267665&full=1&branch=mozilla-inbound#error0 # we don't have branding yet, but we need it. Call it explicitly make[8]: Entering directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales' /tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/JarMaker.py \ -j ../../../../../dist/bin/chrome \ -t /builds/slave/m-in-andrd-xul/build -f flat -c /builds/slave/m-in-andrd-xul/build/mobile/branding/nightly/locales/en-US -DNDEBUG -DTRIMMED -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux -DANDROID=1 -DANDROID_VERSION=5 -DCROSS_COMPILE=1 -DMOZ_THUMB2=1 -DHAVE_ARM_SIMD=1 -DHAVE_ARM_NEON=1 -DMOZILLA_VERSION=\"12.0a1\" -DMOZILLA_VERSION_U=12.0a1 -DNO_PW_GECOS=1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_UINT_T=1 -DHAVE_UNAME_DOMAINNAME_FIELD=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MALLOC_H=1 -DHAVE_SYS_STATFS_H=1 -DHAVE_SYS_SYSMACROS_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLOPEN=1 -DHAVE_MEMMEM=1 -DNO_X11=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DMALLOC_H=\<malloc.h\> -DHAVE_STRNDUP=1 -DHAVE_MEMALIGN=1 -DHAVE_VALLOC=1 -DHAVE_I18N_LC_MESSAGES=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DMOZ_BUILD_APP=mobile/xul -DMOZ_WIDGET_ANDROID=1 -DMOZ_PDF_PRINTING=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DUSE_ARM_KUSER=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_RAW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_ARM_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_TREMOR=1 -DMOZ_XTF=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_USE_NATIVE_POPUP_WINDOWS=1 -DMOZ_TREE_FREETYPE=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_DISABLE_DOMCRYPTO=1 -DMOZ_FEEDS=1 -DMOZ_GFX_OPTIMIZE_MOBILE=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DSIZEOF_INT_P=4 -DMOZ_MEMORY_SIZEOF_PTR_2POW=2 -DMOZ_MEMORY=1 -DMOZ_MEMORY_LINUX=1 -DMOZ_MEMORY_ANDROID=1 -DHAVE_JEMALLOC_VALLOC=1 -DHAVE_JEMALLOC_POSIX_MEMALIGN=1 -DHAVE_JEMALLOC_MEMALIGN=1 -DJSGC_INCREMENTAL=1 -DJS_CRASH_DIAGNOSTICS=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_GRAPHITE=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_COMPONENT_INCLUDE=\"nsBrowserComponents.h\" -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"12.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"12.0a1\" -DFIREFOX_VERSION=12.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_DLL_SUFFIX=\".so\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1 \ /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt package -f -M AndroidManifest.xml -I /tools/android-sdk-r13/platforms/android-13/android.jar -S res -J . --custom-package org.mozilla.gecko /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt: /usr/lib/libz.so.1: no version information available (required by /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt) processing /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn make[8]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales' /tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/Preprocessor.py -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux \ -DBRANDPATH="/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales/../../../dist/bin/chrome/en-US/locale/branding/brand.dtd" \ -DSTRINGSPATH="/builds/slave/m-in-andrd-xul/build/embedding/android/locales/en-US/android_strings.dtd" \ /builds/slave/m-in-andrd-xul/build/embedding/android/locales/../strings.xml.in \ > ../res/values/strings.xml res/values/strings.xml:1: error: Error parsing XML: no element found res/layout/crash_reporter.xml:7: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_message'). res/layout/crash_reporter.xml:13: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_help_message'). res/layout/crash_reporter.xml:18: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_send_report_message'). res/layout/crash_reporter.xml:23: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_include_url'). res/layout/crash_reporter.xml:33: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_close_label'). res/layout/crash_reporter.xml:40: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_restart_label'). make[6]: *** [R.java] Error 1 make[6]: *** Waiting for unfinished jobs.... make[7]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales' make[6]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android' make[5]: *** [libs] Error 2 make[5]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding' make[4]: *** [libs_tier_platform] Error 2 make[4]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[3]: *** [tier_platform] Error 2 make[3]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[2]: *** [default] Error 2 make[2]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[1]: Leaving directory `/builds/slave/m-in-andrd-xul/build' make[1]: *** [realbuild] Error 2 make: *** [build] Error 2 program finished with exit code 2 Because it's Android, we just retrigger, sometimes pretending that a clobber will do something, but it would be nicer if we didn't have a build which constantly fails. https://tbpl.mozilla.org/php/getParsedLog.php?id=8265160&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8265779&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8265860&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8264177&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8234905&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8267755&tree=Mozilla-Inbound
This is kinda the same problem as bug 708437. Which we don't understand, really. It calls the preprocessor to create strings.xml, and it seems like we're so parallel that it's reading the created file before it was finished writing. At least that seems to be the only thing left in my mind. Kyle and I stare at Ted in bug 708437, too. Next thing we could try is to create the preprocessor output in a temporary file, and then mv it to the location of the target.
Attached patch patch (obsolete) — — Splinter Review
this is a similar patch to what I did in bug 712380. I haven't seen bug 708437 since that landed, so hopefully this will do similar black magic voodoo here.
Attachment #585228 - Flags: review?(khuey)
Phil, mind filing a blocked bug to start trees? This bug has a patch and possibly useful comments, star comments are just making the actual bug harder to fix.
do we know when this started happening?
https://tbpl.mozilla.org/php/getParsedLog.php?id=8276641&tree=Firefox Yes, I do mind moving to another bug, this is the bug I filed. My assumption was it started with http://hg.mozilla.org/mozilla-central/rev/b230ea62de17#l1.54 from bug 708015 - it landed last weekend, smelling like the sort of thing that would require a clobber, and it's entirely possible that a clobber slows things down enough that we don't race, and actually works. That would be plenty to let us ignore it for a week that didn't have many pushes.
https://hg.mozilla.org/integration/mozilla-inbound/rev/ea1181d6a816 Doesn't seem to have made any difference, first time failure on the push and the next two: https://tbpl.mozilla.org/php/getParsedLog.php?id=8290286&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8290524&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8292019&tree=Mozilla-Inbound I set yet another clobber, to get some breathing room, so we'll probably see some green now (and who knows, the build system's mostly made of magic, maybe it'll just start working perfectly after that).
Whiteboard: [inbound][leave-open-after-inbound-merge]
https://tbpl.mozilla.org/php/getParsedLog.php?id=8300275&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8300529&tree=Mozilla-Inbound Got a pretty nice bit of peace out of that morning clobber, but I'm starting to think about -j1 until someone gets a handle on it, assuming -j1 will stop us racing.
https://tbpl.mozilla.org/php/getParsedLog.php?id=8320197&tree=Mozilla-Aurora - I wonder how long it would take Aurora to start failing.
Attached patch Patch (obsolete) — — Splinter Review
After talking to ted I think this is the best approach. What happens is that embedding/android has a custom libs rule that it wants to run after all of the standard libs rules complete. There's no way to set up that dependency in the current build system though. Instead, we use the directory ordering to enforce this dependency. The canonical way to do this is with a build/ subdir, but I think it's better not to move everything we have sitting around. Blassey, can you test this patch and confirm that fennec still builds with it?
Attachment #585228 - Attachment is obsolete: true
Attachment #586086 - Flags: review?(blassey.bugs)
Comment on attachment 586086 [details] [diff] [review] Patch no dice flyingfox:objdir-droid-xul blassey$ make -C embedding/android/ /usr/bin/make export make[1]: Nothing to be done for `export'. /usr/bin/make libs make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'. Stop. make: *** [default] Error 2
Attachment #586086 - Flags: review?(blassey.bugs) → review-
Er, you have to make -C objdir/embedding ...
Attachment #586086 - Flags: review- → review?(blassey.bugs)
I'm testing by doing: $> make -C ../objdir-droid-xul/embedding/android/ clean $> make -C ../objdir-droid-xul/embedding/android/ and I get: make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'. Stop. which I now realize just plain won't work with this patch, but is a fairly common work flow when working on the java code
attachment 579864 [details] [diff] [review] in bug 708437 (which is the corresponding bug for mobile/android/base) allows to do that still, by creating a rule for strings.xml that explicitly calls into locales, and doesn't rely on DIRS to be done. Which still seems to be the underlying weirdness, do we know if that's on purpose?
Attachment #586086 - Flags: review?(blassey.bugs) → review-
Resuming posting the logs here, since whilst the comment 18 landing has reduced the frequency, this issue still occurs: https://tbpl.mozilla.org/php/getParsedLog.php?id=8408504&tree=Firefox
Whiteboard: [inbound][leave-open-after-inbound-merge]
The underlying weirdness here is that this makefile wants to run custom libs targets after all the builtin targets are completed (since looping over the subdirectories is the last thing the builtin targets do). The mechanism for that synchronization already exists, it's the subdirectory system. If you don't want to build android/locales from embedding/ then you need to put the relevant things in an android/build directory or something similar.
Summary: This Makefile is broken. You can take khuey's patch, or you can rework the Makefile to put the relevant rules in another subdir. We're not going to take a hacky band-aid patch to work around the brokenness.
OK, thanks for the clarification. Also makes me think that attachment 579864 [details] [diff] [review] is the right way to fix the Makefiles, and should be ported to embedding/android, too. review nag :-)
No, that's the band-aid. The ordering of DIRS should take care of those things if Makefiles are written correctly.
If one could explicitly order the current dir, I'd buy that.
Our infrastructure makes silly assumptions like "we won't leave a platform failing to build for more than a week," so this is now doing fun things like causing us to not get nightly builds. I'm pretty sure I know what caused this, and the right thing to do would be to back that out. Shall I?
Severity: normal → blocker
Priority: -- → P1
Backing out bug 708015 is not an option.
https://tbpl.mozilla.org/php/getParsedLog.php?id=8439087&tree=Mozilla-Inbound
Summary: Frequent Android (XUL-only?) build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found" → Frequent Android XUL build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found"
Attached patch patch — — Splinter Review
Assignee: nobody → blassey.bugs
Attachment #586086 - Attachment is obsolete: true
Attachment #587514 - Flags: review?(khuey)
Comment on attachment 587514 [details] [diff] [review] patch Review of attachment 587514 [details] [diff] [review]: ----------------------------------------------------------------- Ted gave the equivalent patch r- in Bug 708437.
Attachment #587514 - Flags: review?(khuey) → review-
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #70) > Comment on attachment 587514 [details] [diff] [review] > patch > > Review of attachment 587514 [details] [diff] [review]: > ----------------------------------------------------------------- > > Ted gave the equivalent patch r- in Bug 708437. any reason
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #72) > See comments 31-33 in this bug. read them. I guess I disagree with this being a band-aid now that I understand the cause.
https://tbpl.mozilla.org/php/getParsedLog.php?id=8471773&tree=Mozilla-Inbound (the 5th attempt to build for that push, but I accidentally triggered the 6th one at the same time, so maybe it'll manage to build)
This ensures proper dependency: chrome is the rule that goes through subdirectories and ends up creating res/values/strings.xml. This is what needs to be depended upon instead of res/values/strings.xml.
Attachment #587627 - Flags: review?(ted.mielczarek)
Blocks: 717522
OS: Linux → Android
Hardware: x86 → ARM
Landed attachment 587627 [details] [diff] [review] in https://hg.mozilla.org/integration/mozilla-inbound/rev/ec8ded10b50c because you'll like me doing that better than you'll like the alternatives.
Attachment #587514 - Flags: review?(ted.mielczarek) → review-
Comment on attachment 587627 [details] [diff] [review] Avoid race condition for objdir/embedding/android/R.java dependencies Review of attachment 587627 [details] [diff] [review]: ----------------------------------------------------------------- Post-facto review. Whatever. This is still broken but this is the least-shitty solution I guess since nobody actually wants to rewrite the Makefiles to make them correct.
Attachment #587627 - Flags: review?(ted.mielczarek) → review+
Given how things are now, i.e., no more tbpl noise on central, should we mark this bug fixed, and request approval for the hack to be backported to aurora?
That's pretty funny, "tbpl noise," when you very obviously never ever look at the tree itself.
(In reply to Phil Ringnalda (:philor) from comment #152) > That's pretty funny, "tbpl noise," when you very obviously never ever look > at the tree itself. what does that mean?
Assignee: blassey.bugs → nobody
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: