browser/base and browser/devtools are racing to make jars

RESOLVED FIXED in mozilla8

Status

()

Core
Build Config
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: Ehsan, Assigned: rc)

Tracking

Trunk
mozilla8
x86
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1312989362.1312998829.10876.gz&fulltext=1

/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
/builds/slave/m-cen-osx64/build/obj-firefox/x86_64/config/nsinstall -L /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/browser/base -m 644 /builds/slave/m-cen-osx64/build/browser/base/content/openLocationLastURL.jsm /builds/slave/m-cen-osx64/build/browser/base/content/NetworkPrioritizer.jsm /builds/slave/m-cen-osx64/build/browser/base/content/domplate.jsm ../../dist/bin/modules
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/config/JarMaker.py \
	   -j ../../dist/bin/chrome \
	  -t /builds/slave/m-cen-osx64/build -f symlink  -DNDEBUG -DTRIMMED -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin -DMOZ_APP_VERSION=8.0a1 -DAPP_LICENSE_BLOCK=/builds/slave/m-cen-osx64/build/browser/base/content/overrides/app-license.html -DPRE_RELEASE_SUFFIX=""""  -DHAVE_SHELL_SERVICE=1 -DCONTEXT_COPY_IMAGE_CONTENTS=1 -DX_DISPLAY_MISSING=1 -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"8.0a1\" -DMOZILLA_VERSION_U=8.0a1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DXP_MACOSX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X64_ASM=1 -DMOZ_SMIL=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"8.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"8.0a1\" -DFIREFOX_VERSION=8.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1  \
	  /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/base/jar.mn
processing /builds/slave/m-cen-osx64/build/browser/devtools/jar.mn
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 516, in <module>
    main()
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 513, in main
    localedirs=options.l10n_src)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 298, in makeJars
    jardir=jardir)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 243, in makeJar
    localedirs)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 364, in processJarSection
    outHelper, jf)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 413, in _processEntryLine
    outHelper.symlink(realsrc, out)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 467, in symlink
    out = self.ensureDirFor(dest)
  File "/builds/slave/m-cen-osx64/build/config/JarMaker.py", line 459, in ensureDirFor
    os.makedirs(outdir)
  File "/tools/buildbot/bin/../lib/python2.6/os.py", line 157, in makedirs
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx64/build/obj-firefox/x86_64/dist/sdk/bin/xpt.py link _xpidlgen/fuel.xpt _xpidlgen/fuelIApplication.xpt
    mkdir(name, mode)
OSError: [Errno 17] File exists: '../../dist/bin/chrome/browser/content/browser'
NEXT ERROR make[7]: *** [libs] Error 1

catlee thinks that this is a race in xpt.py.  I could believe him.  Kyle, can you please take a look?
That seems ... unlikely.  The traceback is in JarMaker.py ...

Comment 2

6 years ago
Do we have parallel dirs running jarmaker?
Looks like browser/base and browser/devtools are racing.
Summary: Intermittent build failure seemingly because of dependency problems → browser/base and browser/devtools are racing to make jars
(Assignee)

Comment 4

6 years ago
proposing we move browser/base/jar.mn up a level and append the contents of devtools/jar.mn to that. We'll have to prepend base and devtools to the directories in the resultant file.
(Assignee)

Comment 5

6 years ago
Created attachment 552230 [details] [diff] [review]
jar fix

first patch.
Assignee: nobody → rcampbell
Status: NEW → ASSIGNED
Attachment #552230 - Flags: review?(dolske)
(Assignee)

Comment 6

6 years ago
oh teehee. Disregard that patch... Lemme try that again.
(Assignee)

Updated

6 years ago
Attachment #552230 - Attachment is obsolete: true
Attachment #552230 - Flags: review?(dolske)
(Assignee)

Comment 7

6 years ago
this looks like it'll have the potential to break some of our build scripts. Might require moving some defines out of the makefile in browser/base up to browser.

We'll want to check out what's depending on those before any fix can land.
(Assignee)

Comment 8

6 years ago
after correcting this patch, need to reregister the chrome files correctly. Needs work...
(Assignee)

Comment 9

6 years ago
Created attachment 552241 [details] [diff] [review]
jar 2 wip

breaking still. Probably still some paths that need to be tweaked.
(Assignee)

Comment 10

6 years ago
at the risk of further polluting this bug with hare-brained schemes of jar.mn shenanigans, why don't we just move the contents of browser/devtools/jar.mn into browser/base/jar.mn and leave the latter where it is? Deleting the entry under devtools, of course.

No additional build breakage will ensue.

Comment 11

6 years ago
The other solution would be to just not add devtools to PARALLEL_DIRS but DIRS in browser/Makefile.in.
(Assignee)

Comment 12

6 years ago
ooh, that's even better. I'll do that!
(Assignee)

Comment 13

6 years ago
Created attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix
Attachment #552241 - Attachment is obsolete: true
Attachment #552281 - Flags: review?(dolske)
Attachment #552281 - Flags: review?(dolske) → review+
(Assignee)

Comment 14

6 years ago
Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/integration/fx-team/rev/46cd0808d1c5
Attachment #552281 - Attachment description: simpler fix → [in-fx-team] simpler fix
(Assignee)

Updated

6 years ago
Whiteboard: [fixed-in-fx-team]
(Assignee)

Comment 15

6 years ago
had to back this out as it was apparently the cause of bug 679301.

Local tests confirmed that contents of devtools/scratchpad were not being added to the jar at test time. Need to figure out how to do this properly.

http://hg.mozilla.org/integration/fx-team/rev/87995d2de00c
(Assignee)

Updated

6 years ago
Whiteboard: [fixed-in-fx-team] → [backed-out]

Comment 16

6 years ago
PS: I've been thinking about this the other day, but forgot to update the bug:

JarMaker.py does quite some effort to synchronize writing to the same jar, as did it precessor.
Neither made any effort to guard against synchronous writes for flat packaging, as that was out of fashion at the time.

I'm not sure how we'd fight race conditions right for flat packaging. Surely we wouldn't want to create a lock for each file?

Really depends on what we want to guard against, and what's OK to just mess up if people mess up. Say, having parallel writes to the same packaged file shouldn't happen anyway, so not guard against that?

Maybe we should just catch the "file exists" error and verify that it's a directory afterwards?
"app" needs to be the last dir built in browser/, so that things get bundled correctly on Mac.
(Assignee)

Comment 18

6 years ago
Comment on attachment 552281 [details] [diff] [review]
[in-fx-team] simpler fix

http://hg.mozilla.org/mozilla-central/rev/b44e9053d17d
(Assignee)

Updated

6 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Whiteboard: [backed-out]
Target Milestone: --- → mozilla8
(Assignee)

Comment 19

6 years ago
Caught this reply in bugmail after I posted the fix.

(In reply to Axel Hecht [:Pike] from comment #16)
> PS: I've been thinking about this the other day, but forgot to update the
> bug:
> 
> JarMaker.py does quite some effort to synchronize writing to the same jar,
> as did it precessor.
> Neither made any effort to guard against synchronous writes for flat
> packaging, as that was out of fashion at the time.
> 
> I'm not sure how we'd fight race conditions right for flat packaging. Surely
> we wouldn't want to create a lock for each file?
> 
> Really depends on what we want to guard against, and what's OK to just mess
> up if people mess up. Say, having parallel writes to the same packaged file
> shouldn't happen anyway, so not guard against that?
> 
> Maybe we should just catch the "file exists" error and verify that it's a
> directory afterwards?

Sounds like this is worthy of a follow-up bug on JarMaker.py. Would you like me to file it or can you? It sounds like you understand the problem in greater depth than I do.
You need to log in before you can comment on or make changes to this bug.