Last Comment Bug 678021 - browser/base and browser/devtools are racing to make jars
: browser/base and browser/devtools are racing to make jars
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Mac OS X
: -- normal (vote)
: mozilla8
Assigned To: Rob Campbell [:rc] (:robcee)
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-10 13:10 PDT by :Ehsan Akhgari
Modified: 2011-08-17 09:07 PDT (History)
8 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
jar fix (13.23 KB, patch)
2011-08-10 15:02 PDT, Rob Campbell [:rc] (:robcee)
no flags Details | Diff | Splinter Review
jar 2 wip (14.43 KB, patch)
2011-08-10 15:31 PDT, Rob Campbell [:rc] (:robcee)
no flags Details | Diff | Splinter Review
[in-fx-team] simpler fix (481 bytes, patch)
2011-08-10 17:46 PDT, Rob Campbell [:rc] (:robcee)
dolske: review+
Details | Diff | Splinter Review

Description :Ehsan Akhgari 2011-08-10 13:10:04 PDT
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1312989362.1312998829.10876.gz&fulltext=1

/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
/builds/slave/m-cen-osx64/build/obj-firefox/x86_64/config/nsinstall -L /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/browser/base -m 644 /builds/slave/m-cen-osx64/build/browser/base/content/openLocationLastURL.jsm /builds/slave/m-cen-osx64/build/browser/base/content/NetworkPrioritizer.jsm /builds/slave/m-cen-osx64/build/browser/base/content/domplate.jsm ../../dist/bin/modules
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DMOZ_APP_VERSION=8.0a1 -DAPP_LICENSE_BLOCK=/builds/slave/m-cen-osx64/build/browser/base/content/overrides/app-license.html -DPRE_RELEASE_SUFFIX=""""  -DHAVE_SHELL_SERVICE=1 -DCONTEXT_COPY_IMAGE_CONTENTS=1 -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 516, in <module>
    main()
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 513, in main
    localedirs=options.l10n_src)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 298, in makeJars
    jardir=jardir)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 243, in makeJar
    localedirs)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 364, in processJarSection
    outHelper, jf)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 413, in _processEntryLine
    outHelper.symlink(realsrc, out)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 467, in symlink
    out = self.ensureDirFor(dest)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 459, in ensureDirFor
    os.makedirs(outdir)
  File "/tools/buildbot/bin/../lib/python2.6/os.py", line 157, in makedirs
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/dist/sdk/bin/xpt.py link _xpidlgen/fuel.xpt _xpidlgen/fuelIApplication.xpt
    mkdir(name, mode)
OSError: [Errno 17] File exists: '../../dist/bin/chrome/browser/content/browser'
NEXT ERROR make[7]: *** [libs] Error 1

catlee thinks that this is a race in xpt.py.  I could believe him.  Kyle, can you please take a look?
Comment 1 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-08-10 13:56:19 PDT
That seems ... unlikely.  The traceback is in JarMaker.py ...
Comment 2 Axel Hecht [:Pike] 2011-08-10 14:36:05 PDT
Do we have parallel dirs running jarmaker?
Comment 3 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-08-10 14:38:30 PDT
Looks like browser/base and browser/devtools are racing.
Comment 4 Rob Campbell [:rc] (:robcee) 2011-08-10 14:52:00 PDT
proposing we move browser/base/jar.mn up a level and append the contents of devtools/jar.mn to that. We'll have to prepend base and devtools to the directories in the resultant file.
Comment 5 Rob Campbell [:rc] (:robcee) 2011-08-10 15:02:02 PDT
Created attachment 552230 [details] [diff] [review]
jar fix

first patch.
Comment 6 Rob Campbell [:rc] (:robcee) 2011-08-10 15:03:52 PDT
oh teehee. Disregard that patch... Lemme try that again.
Comment 7 Rob Campbell [:rc] (:robcee) 2011-08-10 15:13:16 PDT
this looks like it'll have the potential to break some of our build scripts. Might require moving some defines out of the makefile in browser/base up to browser.

We'll want to check out what's depending on those before any fix can land.
Comment 8 Rob Campbell [:rc] (:robcee) 2011-08-10 15:28:41 PDT
after correcting this patch, need to reregister the chrome files correctly. Needs work...
Comment 9 Rob Campbell [:rc] (:robcee) 2011-08-10 15:31:26 PDT
Created attachment 552241 [details] [diff] [review]
jar 2 wip

breaking still. Probably still some paths that need to be tweaked.
Comment 10 Rob Campbell [:rc] (:robcee) 2011-08-10 15:46:03 PDT
at the risk of further polluting this bug with hare-brained schemes of jar.mn shenanigans, why don't we just move the contents of browser/devtools/jar.mn into browser/base/jar.mn and leave the latter where it is? Deleting the entry under devtools, of course.

No additional build breakage will ensue.
Comment 11 Axel Hecht [:Pike] 2011-08-10 16:18:36 PDT
The other solution would be to just not add devtools to PARALLEL_DIRS but DIRS in browser/Makefile.in.
Comment 12 Rob Campbell [:rc] (:robcee) 2011-08-10 17:39:14 PDT
ooh, that's even better. I'll do that!
Comment 13 Rob Campbell [:rc] (:robcee) 2011-08-10 17:46:45 PDT
Created attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix
Comment 14 Rob Campbell [:rc] (:robcee) 2011-08-15 08:59:07 PDT
Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/integration/fx-team/rev/46cd0808d1c5
Comment 15 Rob Campbell [:rc] (:robcee) 2011-08-16 06:10:27 PDT
had to back this out as it was apparently the cause of bug 679301.

Local tests confirmed that contents of devtools/scratchpad were not being added to the jar at test time. Need to figure out how to do this properly.

http://hg.mozilla.org/integration/fx-team/rev/87995d2de00c
Comment 16 Axel Hecht [:Pike] 2011-08-16 06:18:44 PDT
PS: I've been thinking about this the other day, but forgot to update the bug:

JarMaker.py does quite some effort to synchronize writing to the same jar, as did it precessor.
Neither made any effort to guard against synchronous writes for flat packaging, as that was out of fashion at the time.

I'm not sure how we'd fight race conditions right for flat packaging. Surely we wouldn't want to create a lock for each file?

Really depends on what we want to guard against, and what's OK to just mess up if people mess up. Say, having parallel writes to the same packaged file shouldn't happen anyway, so not guard against that?

Maybe we should just catch the "file exists" error and verify that it's a directory afterwards?
Comment 17 :Gavin Sharp [email: gavin@gavinsharp.com] 2011-08-16 08:01:39 PDT
"app" needs to be the last dir built in browser/, so that things get bundled correctly on Mac.
Comment 18 Rob Campbell [:rc] (:robcee) 2011-08-16 08:36:04 PDT
Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/mozilla-central/rev/b44e9053d17d
Comment 19 Rob Campbell [:rc] (:robcee) 2011-08-17 09:07:32 PDT
Caught this reply in bugmail after I posted the fix.

(In reply to Axel Hecht [:Pike] from comment #16)
> PS: I've been thinking about this the other day, but forgot to update the
> bug:
> 
> JarMaker.py does quite some effort to synchronize writing to the same jar,
> as did it precessor.
> Neither made any effort to guard against synchronous writes for flat
> packaging, as that was out of fashion at the time.
> 
> I'm not sure how we'd fight race conditions right for flat packaging. Surely
> we wouldn't want to create a lock for each file?
> 
> Really depends on what we want to guard against, and what's OK to just mess
> up if people mess up. Say, having parallel writes to the same packaged file
> shouldn't happen anyway, so not guard against that?
> 
> Maybe we should just catch the "file exists" error and verify that it's a
> directory afterwards?

Sounds like this is worthy of a follow-up bug on JarMaker.py. Would you like me to file it or can you? It sounds like you understand the problem in greater depth than I do.

Note You need to log in before you can comment on or make changes to this bug.