678021 - browser/base and browser/devtools are racing to make jars

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(no longer active)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1312989362.1312998829.10876.gz&fulltext=1

/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
/builds/slave/m-cen-osx64/build/obj-firefox/x86_64/config/nsinstall -L /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/browser/base -m 644 /builds/slave/m-cen-osx64/build/browser/base/content/openLocationLastURL.jsm /builds/slave/m-cen-osx64/build/browser/base/content/NetworkPrioritizer.jsm /builds/slave/m-cen-osx64/build/browser/base/content/domplate.jsm ../../dist/bin/modules
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DMOZ_APP_VERSION=8.0a1 -DAPP_LICENSE_BLOCK=/builds/slave/m-cen-osx64/build/browser/base/content/overrides/app-license.html -DPRE_RELEASE_SUFFIX=""""  -DHAVE_SHELL_SERVICE=1 -DCONTEXT_COPY_IMAGE_CONTENTS=1 -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 516, in <module>
    main()
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 513, in main
    localedirs=options.l10n_src)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 298, in makeJars
    jardir=jardir)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 243, in makeJar
    localedirs)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 364, in processJarSection
    outHelper, jf)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 413, in _processEntryLine
    outHelper.symlink(realsrc, out)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 467, in symlink
    out = self.ensureDirFor(dest)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 459, in ensureDirFor
    os.makedirs(outdir)
  File "/tools/buildbot/bin/../lib/python2.6/os.py", line 157, in makedirs
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/dist/sdk/bin/xpt.py link _xpidlgen/fuel.xpt _xpidlgen/fuelIApplication.xpt
    mkdir(name, mode)
OSError: [Errno 17] File exists: '../../dist/bin/chrome/browser/content/browser'
NEXT ERROR make[7]: *** [libs] Error 1

catlee thinks that this is a race in xpt.py.  I could believe him.  Kyle, can you please take a look?

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

That seems ... unlikely.  The traceback is in JarMaker.py ...

Comment 2

[Axel Hecht [:Pike]]

Do we have parallel dirs running jarmaker?

Comment 3

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Looks like browser/base and browser/devtools are racing.

Comment 4

[Rob Campbell [:rc] (:robcee)]

proposing we move browser/base/jar.mn up a level and append the contents of devtools/jar.mn to that. We'll have to prepend base and devtools to the directories in the resultant file.

Comment 5

[Rob Campbell [:rc] (:robcee)]

Attachment: jar fix

first patch.

Comment 6

[Rob Campbell [:rc] (:robcee)]

oh teehee. Disregard that patch... Lemme try that again.

Comment 7

[Rob Campbell [:rc] (:robcee)]

this looks like it'll have the potential to break some of our build scripts. Might require moving some defines out of the makefile in browser/base up to browser.

We'll want to check out what's depending on those before any fix can land.

Comment 8

[Rob Campbell [:rc] (:robcee)]

after correcting this patch, need to reregister the chrome files correctly. Needs work...

Comment 9

[Rob Campbell [:rc] (:robcee)]

Attachment: jar 2 wip

breaking still. Probably still some paths that need to be tweaked.

Comment 10

[Rob Campbell [:rc] (:robcee)]

at the risk of further polluting this bug with hare-brained schemes of jar.mn shenanigans, why don't we just move the contents of browser/devtools/jar.mn into browser/base/jar.mn and leave the latter where it is? Deleting the entry under devtools, of course.

No additional build breakage will ensue.

Comment 11

[Axel Hecht [:Pike]]

The other solution would be to just not add devtools to PARALLEL_DIRS but DIRS in browser/Makefile.in.

Comment 12

[Rob Campbell [:rc] (:robcee)]

ooh, that's even better. I'll do that!

Comment 13

[Rob Campbell [:rc] (:robcee)]

Attachment: [in-fx-team] simpler fix

Comment 14

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/integration/fx-team/rev/46cd0808d1c5

Comment 15

[Rob Campbell [:rc] (:robcee)]

had to back this out as it was apparently the cause of bug 679301.

Local tests confirmed that contents of devtools/scratchpad were not being added to the jar at test time. Need to figure out how to do this properly.

http://hg.mozilla.org/integration/fx-team/rev/87995d2de00c

Comment 16

[Axel Hecht [:Pike]]

PS: I've been thinking about this the other day, but forgot to update the bug:

JarMaker.py does quite some effort to synchronize writing to the same jar, as did it precessor.
Neither made any effort to guard against synchronous writes for flat packaging, as that was out of fashion at the time.

I'm not sure how we'd fight race conditions right for flat packaging. Surely we wouldn't want to create a lock for each file?

Really depends on what we want to guard against, and what's OK to just mess up if people mess up. Say, having parallel writes to the same packaged file shouldn't happen anyway, so not guard against that?

Maybe we should just catch the "file exists" error and verify that it's a directory afterwards?

Comment 17

[:Gavin Sharp [email: [e-mail]]]

"app" needs to be the last dir built in browser/, so that things get bundled correctly on Mac.

Comment 18

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/mozilla-central/rev/b44e9053d17d

Comment 19

[Rob Campbell [:rc] (:robcee)]

Caught this reply in bugmail after I posted the fix.

(In reply to Axel Hecht [:Pike] from comment #16)
> PS: I've been thinking about this the other day, but forgot to update the
> bug:
> 
> JarMaker.py does quite some effort to synchronize writing to the same jar,
> as did it precessor.
> Neither made any effort to guard against synchronous writes for flat
> packaging, as that was out of fashion at the time.
> 
> I'm not sure how we'd fight race conditions right for flat packaging. Surely
> we wouldn't want to create a lock for each file?
> 
> Really depends on what we want to guard against, and what's OK to just mess
> up if people mess up. Say, having parallel writes to the same packaged file
> shouldn't happen anyway, so not guard against that?
> 
> Maybe we should just catch the "file exists" error and verify that it's a
> directory afterwards?

Sounds like this is worthy of a follow-up bug on JarMaker.py. Would you like me to file it or can you? It sounds like you understand the problem in greater depth than I do.