Frequent Android XUL build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found"

RESOLVED FIXED

Status

P1
blocker
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: philor, Unassigned)

Tracking

Trunk
ARM
Android
Dependency tree / graph

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

7 years ago
e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=8267665&full=1&branch=mozilla-inbound#error0

# we don't have branding yet, but we need it. Call it explicitly
make[8]: Entering directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales'
/tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/JarMaker.py \
	   -j ../../../../../dist/bin/chrome \
	  -t /builds/slave/m-in-andrd-xul/build -f flat  -c /builds/slave/m-in-andrd-xul/build/mobile/branding/nightly/locales/en-US -DNDEBUG -DTRIMMED -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux -DANDROID=1 -DANDROID_VERSION=5 -DCROSS_COMPILE=1 -DMOZ_THUMB2=1 -DHAVE_ARM_SIMD=1 -DHAVE_ARM_NEON=1 -DMOZILLA_VERSION=\"12.0a1\" -DMOZILLA_VERSION_U=12.0a1 -DNO_PW_GECOS=1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_UINT_T=1 -DHAVE_UNAME_DOMAINNAME_FIELD=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MALLOC_H=1 -DHAVE_SYS_STATFS_H=1 -DHAVE_SYS_SYSMACROS_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLOPEN=1 -DHAVE_MEMMEM=1 -DNO_X11=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DMALLOC_H=\<malloc.h\> -DHAVE_STRNDUP=1 -DHAVE_MEMALIGN=1 -DHAVE_VALLOC=1 -DHAVE_I18N_LC_MESSAGES=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DMOZ_BUILD_APP=mobile/xul -DMOZ_WIDGET_ANDROID=1 -DMOZ_PDF_PRINTING=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DUSE_ARM_KUSER=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_RAW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_ARM_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_TREMOR=1 -DMOZ_XTF=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_USE_NATIVE_POPUP_WINDOWS=1 -DMOZ_TREE_FREETYPE=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_DISABLE_DOMCRYPTO=1 -DMOZ_FEEDS=1 -DMOZ_GFX_OPTIMIZE_MOBILE=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DSIZEOF_INT_P=4 -DMOZ_MEMORY_SIZEOF_PTR_2POW=2 -DMOZ_MEMORY=1 -DMOZ_MEMORY_LINUX=1 -DMOZ_MEMORY_ANDROID=1 -DHAVE_JEMALLOC_VALLOC=1 -DHAVE_JEMALLOC_POSIX_MEMALIGN=1 -DHAVE_JEMALLOC_MEMALIGN=1 -DJSGC_INCREMENTAL=1 -DJS_CRASH_DIAGNOSTICS=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_GRAPHITE=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_COMPONENT_INCLUDE=\"nsBrowserComponents.h\" -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"12.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"12.0a1\" -DFIREFOX_VERSION=12.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_DLL_SUFFIX=\".so\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn
/tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt package -f -M AndroidManifest.xml -I /tools/android-sdk-r13/platforms/android-13/android.jar -S res -J . --custom-package org.mozilla.gecko
/tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt: /usr/lib/libz.so.1: no version information available (required by /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt)
processing /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn
make[8]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales'
/tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/Preprocessor.py -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux \
	  -DBRANDPATH="/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales/../../../dist/bin/chrome/en-US/locale/branding/brand.dtd" \
	  -DSTRINGSPATH="/builds/slave/m-in-andrd-xul/build/embedding/android/locales/en-US/android_strings.dtd" \
	  /builds/slave/m-in-andrd-xul/build/embedding/android/locales/../strings.xml.in \
	  > ../res/values/strings.xml
res/values/strings.xml:1: error: Error parsing XML: no element found
res/layout/crash_reporter.xml:7: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_message').
res/layout/crash_reporter.xml:13: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_help_message').
res/layout/crash_reporter.xml:18: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_send_report_message').
res/layout/crash_reporter.xml:23: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_include_url').
res/layout/crash_reporter.xml:33: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_close_label').
res/layout/crash_reporter.xml:40: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_restart_label').
make[6]: *** [R.java] Error 1
make[6]: *** Waiting for unfinished jobs....
make[7]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales'
make[6]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android'
make[5]: *** [libs] Error 2
make[5]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding'
make[4]: *** [libs_tier_platform] Error 2
make[4]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox'
make[3]: *** [tier_platform] Error 2
make[3]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox'
make[1]: Leaving directory `/builds/slave/m-in-andrd-xul/build'
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
program finished with exit code 2

Because it's Android, we just retrigger, sometimes pretending that a clobber will do something, but it would be nicer if we didn't have a build which constantly fails.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8265160&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=8265779&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=8265860&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=8264177&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=8234905&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=8267755&tree=Mozilla-Inbound

Comment 1

7 years ago
This is kinda the same problem as bug 708437.

Which we don't understand, really. It calls the preprocessor to create strings.xml, and it seems like we're so parallel that it's reading the created file before it was finished writing. At least that seems to be the only thing left in my mind.

Kyle and I stare at Ted in bug 708437, too.

Next thing we could try is to create the preprocessor output in a temporary file, and then mv it to the location of the target.
Created attachment 585228 [details] [diff] [review]
patch

this is a similar patch to what I did in bug 712380. I haven't seen bug 708437 since that landed, so hopefully this will do similar black magic voodoo here.
Attachment #585228 - Flags: review?(khuey)

Comment 9

7 years ago
Phil, mind filing a blocked bug to start trees? This bug has a patch and possibly useful comments, star comments are just making the actual bug harder to fix.
do we know when this started happening?
(Reporter)

Comment 11

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=8276641&tree=Firefox

Yes, I do mind moving to another bug, this is the bug I filed.

My assumption was it started with http://hg.mozilla.org/mozilla-central/rev/b230ea62de17#l1.54 from bug 708015 - it landed last weekend, smelling like the sort of thing that would require a clobber, and it's entirely possible that a clobber slows things down enough that we don't race, and actually works. That would be plenty to let us ignore it for a week that didn't have many pushes.
(Reporter)

Comment 14

7 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/ea1181d6a816

Doesn't seem to have made any difference, first time failure on the push and the next two:

https://tbpl.mozilla.org/php/getParsedLog.php?id=8290286&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=8290524&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=8292019&tree=Mozilla-Inbound

I set yet another clobber, to get some breathing room, so we'll probably see some green now (and who knows, the build system's mostly made of magic, maybe it'll just start working perfectly after that).
Whiteboard: [inbound][leave-open-after-inbound-merge]
(Reporter)

Comment 15

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=8300275&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=8300529&tree=Mozilla-Inbound

Got a pretty nice bit of peace out of that morning clobber, but I'm starting to think about -j1 until someone gets a handle on it, assuming -j1 will stop us racing.
(Reporter)

Comment 19

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=8320197&tree=Mozilla-Aurora - I wonder how long it would take Aurora to start failing.
Created attachment 586086 [details] [diff] [review]
Patch

After talking to ted I think this is the best approach.  What happens is that embedding/android has a custom libs rule that it wants to run after all of the standard libs rules complete.  There's no way to set up that dependency in the current build system though.

Instead, we use the directory ordering to enforce this dependency.  The canonical way to do this is with a build/ subdir, but I think it's better not to move everything we have sitting around.

Blassey, can you test this patch and confirm that fennec still builds with it?
Attachment #585228 - Attachment is obsolete: true
Attachment #586086 - Flags: review?(blassey.bugs)
Comment on attachment 586086 [details] [diff] [review]
Patch

no dice

flyingfox:objdir-droid-xul blassey$ make -C embedding/android/ 
/usr/bin/make export
make[1]: Nothing to be done for `export'.
/usr/bin/make libs
make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'.  Stop.
make: *** [default] Error 2
Attachment #586086 - Flags: review?(blassey.bugs) → review-
Er, you have to make -C objdir/embedding ...
I'm testing by doing:
$> make -C ../objdir-droid-xul/embedding/android/ clean
$> make -C ../objdir-droid-xul/embedding/android/ 

and I get:
make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'.  Stop.

which I now realize just plain won't work with this patch, but is a fairly common work flow when working on the java code

Comment 28

7 years ago
attachment 579864 [details] [diff] [review] in bug 708437 (which is the corresponding bug for mobile/android/base) allows to do that still, by creating a rule for strings.xml that explicitly calls into locales, and doesn't rely on DIRS to be done.

Which still seems to be the underlying weirdness, do we know if that's on purpose?
Attachment #586086 - Flags: review?(blassey.bugs) → review-
Resuming posting the logs here, since whilst the comment 18 landing has reduced the frequency, this issue still occurs:
https://tbpl.mozilla.org/php/getParsedLog.php?id=8408504&tree=Firefox
Whiteboard: [inbound][leave-open-after-inbound-merge]
The underlying weirdness here is that this makefile wants to run custom libs targets after all the builtin targets are completed (since looping over the subdirectories is the last thing the builtin targets do).  The mechanism for that synchronization already exists, it's the subdirectory system.  If you don't want to build android/locales from embedding/ then you need to put the relevant things in an android/build directory or something similar.
Summary: This Makefile is broken. You can take khuey's patch, or you can rework the Makefile to put the relevant rules in another subdir. We're not going to take a hacky band-aid patch to work around the brokenness.

Comment 32

7 years ago
OK, thanks for the clarification. Also makes me think that attachment 579864 [details] [diff] [review] is the right way to fix the Makefiles, and should be ported to embedding/android, too. review nag :-)
No, that's the band-aid. The ordering of DIRS should take care of those things if Makefiles are written correctly.

Comment 34

7 years ago
If one could explicitly order the current dir, I'd buy that.
(Reporter)

Comment 35

7 years ago
Our infrastructure makes silly assumptions like "we won't leave a platform failing to build for more than a week," so this is now doing fun things like causing us to not get nightly builds.

I'm pretty sure I know what caused this, and the right thing to do would be to back that out. Shall I?
Severity: normal → blocker
Priority: -- → P1

Comment 36

7 years ago
Backing out bug 708015 is not an option.
(Reporter)

Comment 47

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=8439087&tree=Mozilla-Inbound
Summary: Frequent Android (XUL-only?) build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found" → Frequent Android XUL build failure starting with "res/values/strings.xml:1: error: Error parsing XML: no element found"
Created attachment 587514 [details] [diff] [review]
patch
Assignee: nobody → blassey.bugs
Attachment #586086 - Attachment is obsolete: true
Attachment #587514 - Flags: review?(khuey)
Comment on attachment 587514 [details] [diff] [review]
patch

Review of attachment 587514 [details] [diff] [review]:
-----------------------------------------------------------------

Ted gave the equivalent patch r- in Bug 708437.
Attachment #587514 - Flags: review?(khuey) → review-
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #70)
> Comment on attachment 587514 [details] [diff] [review]
> patch
> 
> Review of attachment 587514 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Ted gave the equivalent patch r- in Bug 708437.

any reason
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #72)
> See comments 31-33 in this bug.

read them. I guess I disagree with this being a band-aid now that I understand the cause.
(Reporter)

Comment 76

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=8471773&tree=Mozilla-Inbound (the 5th attempt to build for that push, but I accidentally triggered the 6th one at the same time, so maybe it'll manage to build)
Created attachment 587627 [details] [diff] [review]
Avoid race condition for objdir/embedding/android/R.java dependencies

This ensures proper dependency: chrome is the rule that goes through subdirectories and ends up creating res/values/strings.xml. This is what needs to be depended upon instead of res/values/strings.xml.
Attachment #587627 - Flags: review?(ted.mielczarek)

Updated

7 years ago
Blocks: 717522

Updated

7 years ago
OS: Linux → Android
Hardware: x86 → ARM
(Reporter)

Comment 148

7 years ago
Landed attachment 587627 [details] [diff] [review] in https://hg.mozilla.org/integration/mozilla-inbound/rev/ec8ded10b50c because you'll like me doing that better than you'll like the alternatives.
Attachment #587514 - Flags: review?(ted.mielczarek) → review-
Comment on attachment 587627 [details] [diff] [review]
Avoid race condition for objdir/embedding/android/R.java dependencies

Review of attachment 587627 [details] [diff] [review]:
-----------------------------------------------------------------

Post-facto review. Whatever. This is still broken but this is the least-shitty solution I guess since nobody actually wants to rewrite the Makefiles to make them correct.
Attachment #587627 - Flags: review?(ted.mielczarek) → review+

Comment 151

7 years ago
Given how things are now, i.e., no more tbpl noise on central, should we mark this bug fixed, and request approval for the hack to be backported to aurora?
(Reporter)

Comment 152

7 years ago
That's pretty funny, "tbpl noise," when you very obviously never ever look at the tree itself.
(In reply to Phil Ringnalda (:philor) from comment #152)
> That's pretty funny, "tbpl noise," when you very obviously never ever look
> at the tree itself.

what does that mean?
Assignee: blassey.bugs → nobody
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.