Closed Bug 1462493 Opened 2 years ago Closed 2 years ago

When attempting to build with Tup backend: "tup error: Expected to write to file"

Categories

(Firefox Build System :: General, defect)

3 Branch
defect
Not set

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: kmag, Assigned: mshal)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

I've been seeing this ever since I first tried to build with the Tup backend a couple of months ago. I've been ignoring it and checking back every couple of weeks, assuming it would be fixed. But given that Tup builds are running on automation now, I'm assuming it has something to do with my configuration.

Every time I try to build, it gets to about 67%, and then I get a bunch of errors like these:

 1:23.34 *  66%  9866) obj-tup/toolkit/content: Preprocess /home/kris/code/mozilla-central/obj-tup/dist/bin/chrome/toolkit/content/global/bindings/tabbox.xml
 1:23.34  *** tup errors ***
 1:23.34 tup error: Expected to write to file '../../dist/bin/chrome/toolkit/content/global/bindings/tabbox.xml' from cmd 768048 but didn't
 1:23.34  *** Command failed due to errors processing the output dependencies.
 1:23.35 *  66%  9867) obj-tup/devtools/client: Preprocess /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/chrome/devtools/content/debugger/debugger.xul
 1:23.35  *** tup errors ***
 1:23.35 tup error: Expected to write to file '../../dist/bin/browser/chrome/devtools/content/debugger/debugger.xul' from cmd 774004 but didn't
 1:23.35  *** Command failed due to errors processing the output dependencies.
 1:23.35 *  66%  9868) obj-tup/devtools/client/webide/content: Preprocess /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/chrome/webide/content/webide.xul
 1:23.35  *** tup errors ***
 1:23.35 tup error: Expected to write to file '../../../../dist/bin/browser/chrome/webide/content/webide.xul' from cmd 774023 but didn't
 1:23.35  *** Command failed due to errors processing the output dependencies.

I've checked, and those files do exist. I was initially assuming it had something to do with FileAvoidWrite confusing Tup by not changing timestamps, but that doesn't make sense for a clobber build, and when I run the commands manually, they do update the timestamps of the files.

In case it helps, the verbose output looks like:

 0:03.48 [ tup ] [2.987s] Executing Commands...
 0:03.60 *   0%    1) obj-tup/browser/base: PYTHONDONTWRITEBYTECODE=1 /home/kris/code/mozilla-central/obj-tup/_virtualenvs/init/bin/python -m mozbuild.action.preprocessor -DNDEBUG=1 -DTRIMMED=1 -DMOZ_APP_VERSION=62.0a1 -DMOZ_APP_VERSION_DISPLAY=62.0a1 -DAPP_LICENSE_BLOCK=/home/kris/code/mozilla-central/browser/base/content/overrides/app-license.html -DCONTEXT_COPY_IMAGE_CONTENTS=1 -DMENUBAR_CAN_AUTOHIDE=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=26 -DATK_REV_VERSION=0 -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=62.0a1 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DGDK_VERSION_MAX_ALLOWED=GDK_VERSION_3_4 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_64BIT=1 -DJS_BUILD_BINAST=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_PUNBOX64=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_UAVERSION='"62.0"' -DMOZILLA_VERSION='"62.0a1"' -DMOZILLA_VERSION_U=62.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ADDON_SIGNING=1 -DMOZ_ALLOW_LEGACY_EXTENSIONS=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"62.0a1"' -DMOZ_AV1=1 -DMOZ_BUILD_APP=browser -DMOZ_BUNDLED_FONTS=1 -DMOZ_CONTENT_SANDBOX=1 -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_GMP_SANDBOX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightly -DMOZ_MEMORY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PROFILING=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_VTUNE=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DAB_CD=en-US -DBOOKMARKS_INCLUDE_DIR=/home/kris/code/mozilla-central/browser/locales/en-US/profile -DBUILD_FASTER=1 /home/kris/code/mozilla-central/browser/base/content/webext-panels.xul -o /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/chrome/browser/content/browser/webext-panels.xul --marker=#
 0:03.61  *** tup errors ***
 0:03.61 tup error: Expected to write to file '../../dist/bin/browser/chrome/browser/content/browser/webext-panels.xul' from cmd 734170 but didn't
 0:03.61  *** Command failed due to errors processing the output dependencies.
 0:03.61 *   0%    2) obj-tup/browser/base: PYTHONDONTWRITEBYTECODE=1 /home/kris/code/mozilla-central/obj-tup/_virtualenvs/init/bin/python -m mozbuild.action.preprocessor -DNDEBUG=1 -DTRIMMED=1 -DMOZ_APP_VERSION=62.0a1 -DMOZ_APP_VERSION_DISPLAY=62.0a1 -DAPP_LICENSE_BLOCK=/home/kris/code/mozilla-central/browser/base/content/overrides/app-license.html -DCONTEXT_COPY_IMAGE_CONTENTS=1 -DMENUBAR_CAN_AUTOHIDE=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=26 -DATK_REV_VERSION=0 -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=62.0a1 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DGDK_VERSION_MAX_ALLOWED=GDK_VERSION_3_4 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_64BIT=1 -DJS_BUILD_BINAST=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_PUNBOX64=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_UAVERSION='"62.0"' -DMOZILLA_VERSION='"62.0a1"' -DMOZILLA_VERSION_U=62.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ADDON_SIGNING=1 -DMOZ_ALLOW_LEGACY_EXTENSIONS=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"62.0a1"' -DMOZ_AV1=1 -DMOZ_BUILD_APP=browser -DMOZ_BUNDLED_FONTS=1 -DMOZ_CONTENT_SANDBOX=1 -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_GMP_SANDBOX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightly -DMOZ_MEMORY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PROFILING=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_VTUNE=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DAB_CD=en-US -DBOOKMARKS_INCLUDE_DIR=/home/kris/code/mozilla-central/browser/locales/en-US/profile -DBUILD_FASTER=1 /home/kris/code/mozilla-central/browser/base/content/pageinfo/pageInfo.xul -o /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/chrome/browser/content/browser/pageinfo/pageInfo.xul --marker=#
 0:03.61  *** tup errors ***
 0:03.61 tup error: Expected to write to file '../../dist/bin/browser/chrome/browser/content/browser/pageinfo/pageInfo.xul' from cmd 734186 but didn't
 0:03.61  *** Command failed due to errors processing the output dependencies.
 0:03.61 *   0%    3) obj-tup/browser/extensions/onboarding: PYTHONDONTWRITEBYTECODE=1 /home/kris/code/mozilla-central/obj-tup/_virtualenvs/init/bin/python -m mozbuild.action.preprocessor -DNDEBUG=1 -DTRIMMED=1 -DMOZ_APP_VERSION=62.0a1 -DMOZ_APP_MAXVERSION=62.0a1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=26 -DATK_REV_VERSION=0 -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=62.0a1 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DGDK_VERSION_MAX_ALLOWED=GDK_VERSION_3_4 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_64BIT=1 -DJS_BUILD_BINAST=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_PUNBOX64=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_UAVERSION='"62.0"' -DMOZILLA_VERSION='"62.0a1"' -DMOZILLA_VERSION_U=62.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ADDON_SIGNING=1 -DMOZ_ALLOW_LEGACY_EXTENSIONS=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"62.0a1"' -DMOZ_AV1=1 -DMOZ_BUILD_APP=browser -DMOZ_BUNDLED_FONTS=1 -DMOZ_CONTENT_SANDBOX=1 -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_GMP_SANDBOX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightly -DMOZ_MEMORY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PROFILING=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_VTUNE=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DAB_CD=en-US -DBOOKMARKS_INCLUDE_DIR=/home/kris/code/mozilla-central/browser/locales/en-US/profile -DBUILD_FASTER=1 /home/kris/code/mozilla-central/browser/extensions/onboarding/install.rdf.in -o /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/features/onboarding@mozilla.org/install.rdf --marker=#
 0:03.61  *** tup errors ***
 0:03.61 tup error: Expected to write to file '../../../dist/bin/browser/features/onboarding@mozilla.org/install.rdf' from cmd 734195 but didn't
(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #0)
>  1:23.35  *** tup errors ***
>  1:23.35 tup error: Expected to write to file
> '../../../../dist/bin/browser/chrome/webide/content/webide.xul' from cmd
> 774023 but didn't
>  1:23.35  *** Command failed due to errors processing the output
> dependencies.
> 
> I've checked, and those files do exist. I was initially assuming it had
> something to do with FileAvoidWrite confusing Tup by not changing
> timestamps, but that doesn't make sense for a clobber build, and when I run
> the commands manually, they do update the timestamps of the files.

Those file writes are supposed to be picked up by tup's internal dependency checking mechanism, but for some reason they're not in your case. Are you using tup from the toolchain task that's used in automation? Or did you install it via some other means? And what platform are you on?
(In reply to Michael Shal [:mshal] from comment #1)
> Those file writes are supposed to be picked up by tup's internal dependency
> checking mechanism, but for some reason they're not in your case. Are you
> using tup from the toolchain task that's used in automation? Or did you
> install it via some other means? And what platform are you on?

I installed the ArchLinux package, which is unpatched tup 0.7.6:

https://www.archlinux.org/packages/community/x86_64/tup/

When I use the version generated by the build-tup-linux-sh script, those errors go away, but I get these instead:

 1:49.10 *  74% 11058) obj-tup/modules/zlib/src: HOST_CC /home/kris/code/mozilla-central/modules/zlib/src/gzwrite.c
 1:49.10 /home/kris/code/mozilla-central/modules/zlib/src/gzwrite.c:89:20: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
 1:49.10             writ = write(state->fd, strm->next_in, put);
 1:49.10                    ^
 1:49.10 /home/kris/code/mozilla-central/modules/zlib/src/gzwrite.c:110:24: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
 1:49.10                 writ = write(state->fd, state->x.next, put);
 1:49.10                        ^
 1:49.10 /home/kris/code/mozilla-central/modules/zlib/src/gzwrite.c:661:9: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
 1:49.10     if (close(state->fd) == -1)
 1:49.10         ^
 1:49.10 3 warnings generated.
 1:49.10  *** tup errors ***
 1:49.10 tup error: Expected to write to file 'host_gzwrite.o' from cmd 997247 but didn't
 1:49.10  *** Command failed due to errors processing the output dependencies.

I suppose that has something to do with -Werror being passed where it shouldn't. I'm still no clearer on the cause of the original errors, though.
(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #2)
> I suppose that has something to do with -Werror being passed where it
> shouldn't.

Hm. Actually, no, that doesn't seem to be it. In verbose mode, I get:

 0:41.24 *   7%  315) obj-tup/modules/zlib/src: /usr/bin/sccache /usr/bin/clang -std=gnu99 -DXP_UNIX -O2 -DNDEBUG=1 -DTRIMMED=1 -I/home/kris/code/mozilla-central/modules/zlib/src -I/home/kris/code/mozilla-central/obj-tup/modules/zlib/src -I/home/kris/code/mozilla-central/obj-tup/dist/include -c /home/kris/code/mozilla-central/modules/zlib/src/gzlib.c -o host_gzlib.o
 0:41.24 /home/kris/code/mozilla-central/modules/zlib/src/gzlib.c:252:9: warning: implicit declaration of function 'lseek' is invalid in C99 [-Wimplicit-function-declaration]
...         ^
 0:41.25 5 warnings generated.
 0:41.25  *** tup errors ***
 0:41.25 tup error: Expected to write to file 'host_gzlib.o' from cmd 997243 but didn't

But the command definitely succeeds, and the object file definitely gets updated...
(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #3)
> (In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're
> blocked) from comment #2)
> > I suppose that has something to do with -Werror being passed where it
> > shouldn't.
> 
> Hm. Actually, no, that doesn't seem to be it. In verbose mode, I get:
> 
>  0:41.24 *   7%  315) obj-tup/modules/zlib/src: /usr/bin/sccache
> /usr/bin/clang -std=gnu99 -DXP_UNIX -O2 -DNDEBUG=1 -DTRIMMED=1
> -I/home/kris/code/mozilla-central/modules/zlib/src
> -I/home/kris/code/mozilla-central/obj-tup/modules/zlib/src
> -I/home/kris/code/mozilla-central/obj-tup/dist/include -c
> /home/kris/code/mozilla-central/modules/zlib/src/gzlib.c -o host_gzlib.o
>  0:41.24 /home/kris/code/mozilla-central/modules/zlib/src/gzlib.c:252:9:
> warning: implicit declaration of function 'lseek' is invalid in C99
> [-Wimplicit-function-declaration]
> ...         ^
>  0:41.25 5 warnings generated.
>  0:41.25  *** tup errors ***
>  0:41.25 tup error: Expected to write to file 'host_gzlib.o' from cmd 997243
> but didn't
> 
> But the command definitely succeeds, and the object file definitely gets
> updated...

Ahh, this error is because sccache does not yet work with tup. The problem is that sccache sends the preprocessed source to a separate server process, and the server process ultimately writes the file. So this essentially bypasses the dependency detection mechanism by writing to files via TCP instead of open(). There is a bunch of work going on in sccache for distributed compilation that may change the behavior of the server process, so I'm planning to test it out with tup after that lands and fix it then if it doesn't already work.

I'm still trying to reproduce the python errors in my arch VM...
(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #0)
> I've been seeing this ever since I first tried to build with the Tup backend
> a couple of months ago. I've been ignoring it and checking back every couple
> of weeks, assuming it would be fixed. But given that Tup builds are running
> on automation now, I'm assuming it has something to do with my configuration.
> 
> Every time I try to build, it gets to about 67%, and then I get a bunch of
> errors like these:
> 
>  1:23.35 *  66%  9868) obj-tup/devtools/client/webide/content: Preprocess
> /home/kris/code/mozilla-central/obj-tup/dist/bin/browser/chrome/webide/
> content/webide.xul
>  1:23.35  *** tup errors ***
>  1:23.35 tup error: Expected to write to file
> '../../../../dist/bin/browser/chrome/webide/content/webide.xul' from cmd
> 774023 but didn't
>  1:23.35  *** Command failed due to errors processing the output
> dependencies.

I was able to reproduce this. So it looks like Arch doesn't allow unprivileged accounts to use user namespaces by default, which is the mechanism tup uses with the FUSE dependency tracker to chroot into a sandbox. Without the user namespace, tup falls back to a non-sandboxed mode which can miss file accesses that use full paths, like this python process that writes to /home/user/.../webide.xul. (There should be a message like "tup warning: unshare(CLONE_NEWUSER) failed, and tup is not privileged. Subprocesses will have '.tup/mnt' paths for the current working directory and some dependencies may be missed." when tup starts, but that's probably buried in the build log). There are a few options here:

1) Tup can work around this by making the tup binary suid - there's a small message about this when installing tup from pacman:

>>> If you want to be able to run commands in chroot-ed environment (e.g. for full dependency tracking)
    set suid bit on tup binary: 'sudo chmod +s /usr/bin/tup'.

2) Enable 'sysctl kernel.unprivileged_userns_clone=1' to allow tup to use user namespaces. This is similar to what's required to run containers on Arch (https://wiki.archlinux.org/index.php/Linux_Containers)

3) Use the binary from the toolchain task, which has tup configured with the LD_PRELOAD dependency tracker instead of the FUSE tracker. This disables some other features in tup, but none of those are required to build mozilla-central. (And as mentioned above, sccache needs to be disabled until we get it working with tup, but that's independent of the dependency tracker).
I see. Thanks.

Would it be worth trying to check for these things during configure, and bail out if we're trying to build with sccache, or we detect that tup won't be able to chroot and isn't built with ldpreload?

I guess it might not be worth the effort while the backend is still experimental, but as more people start using it, more people will probably start running into this.
Keywords: in-triage
Assignee: nobody → mshal
Keywords: in-triage
Depends on: 1377524
Attachment #8980742 - Flags: review?(core-build-config-reviews) → review?(cmanchester)
Comment on attachment 8980742 [details]
Bug 1462493 - Add a check to warn about sccache usage with tup;

https://reviewboard.mozilla.org/r/246910/#review253092
Attachment #8980742 - Flags: review?(cmanchester) → review+
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/87d27981670d
Add a check to warn about sccache usage with tup; r=chmanchester
https://hg.mozilla.org/mozilla-central/rev/87d27981670d
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.