Closed Bug 848406 Opened 12 years ago Closed 12 years ago

Missing Nightly updates for localized builds on Linux 64 if respins of daily builds happen

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 858953

People

(Reporter: whimboo, Unassigned)

References

(
URL
)

Details

(Keywords: regression)

Our Mozmill tests have identified a problem with updates for localized Nightly builds: http://mozmill-ci.blargon7.com/#/update/report/00ba6b3792451db672d192b36831de83 As you can see we are testing updates from yesterdays build to the one built today. When you check the AUS URL it's empty and doesn't list the update to the 20130306* build. https://aus3.mozilla.org/update/3/Firefox/22.0a1/20130305030833/Linux_x86_64-gcc3/fr/nightly/Linux%203.5.0-23-generic%20(GTK%202.24.13)/default/default/update.xml?force=1 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/firefox-22.0a1.fr.linux-x86_64.tar.bz2 In the pulse request we are getting a build id and the previous build id of the same value. That is bug 808550. I tried to reproduce it manually and can confirm that updates are not offered. Steps on a x86-64 system: 1. Download https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-05-03-08-33-mozilla-central-l10n/firefox-22.0a1.fr.linux-x86_64.tar.bz2 2. Unpack and start Firefox 3. Try to update Step 3 doesn't show that any update is available. Not sure how this is related to the appropriate releng bug 832510. I leave it up to you for the decision.
The build machine downloaded 33261672 bytes from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-22.0a1.en-US.linux-x86_64.tar.bz2 Today's build is 33346587 bytes. Yesterday's build at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-03-05-03-08-33-mozilla-central/firefox-22.0a1.en-US.linux-x86_64.tar.bz2 was 33261672 bytes. I suspect that we're seeing some caching issues on ftp causing us to get the previous nightly at the same URL. Today's en-US build was uploaded at 13:20:49 GMT == 05:20:49 PT. This 'fr' repack job started at 05:33:08. The cache control headers from ftp indicate that there's a 1 hour cache on these files. I don't think this is at all related to bug 832510. The problem is that yesterday's nightly is downloaded, and so updates are generated only as far as that.
So there are no builds, tinderbox is all joyfully in flames for linux, and linux only. http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n/1362577450.1362577927.14987.gz&fulltext=1#err0 says DEBUG: doshell: chrootPath:/builds/mock_mozilla/mozilla-centos6-i386/root/, uid:500, gid:494 DEBUG: doshell environment: {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'TZ': 'EST5EDT', 'HOSTNAME': 'mock', 'HOME': '/builds', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'TMPDIR': '/tmp'} DEBUG: doshell: command: /usr/bin/env make DEBUG: child environment: {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'TZ': 'EST5EDT', 'HOSTNAME': 'mock', 'HOME': '/builds', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'TMPDIR': '/tmp'} mkdir -p ".deps/" /tools/gcc-4.5-0moz3/bin/gcc -DMOZ_GLUE_IN_PROGRAM -DNO_NSPR_10_SUPPORT -I../config -I. -I../dist/include -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nspr -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nss -fPIC -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-limits -Wempty-body -Wno-unused -Wcast-align -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -pthread -pipe -DNDEBUG -DTRIMMED -g -O3 -fomit-frame-pointer -include ../mozilla-config.h -DMOZILLA_CLIENT -MD -MF .deps/elf-dynstr-gc.pp -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -o elf-dynstr-gc elf-dynstr-gc.c -lpthread -Wl,-z,noexecstack -Wl,--build-id -Wl,-rpath-link,/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/bin -Wl,-rpath-link,/usr/local/lib -lgobject-2.0 -lglib-2.0 nsinstall.c /tools/gcc-4.5-0moz3/bin/gcc -o host_nsinstall.o -c -DXP_UNIX -O3 -DUNICODE -D_UNICODE -I../config -I. -I../dist/include -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nspr -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nss -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nspr /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/nsinstall.c pathsub.c /tools/gcc-4.5-0moz3/bin/gcc -o host_pathsub.o -c -DXP_UNIX -O3 -DUNICODE -D_UNICODE -I../config -I. -I../dist/include -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nspr -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nss -I/builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/dist/include/nspr /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/pathsub.c /tools/gcc-4.5-0moz3/bin/gcc -o nsinstall_real -DXP_UNIX -O3 -DUNICODE -D_UNICODE host_nsinstall.o host_pathsub.o cp nsinstall_real nsinstall.tmp mv nsinstall.tmp nsinstall /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/_virtualenv/bin/python /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/../config/nsinstall.py -m 644 "nsinstall" "../dist/bin" /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/_virtualenv/bin/python /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/../config/nsinstall.py -m 644 "../mozilla-config.h" "../dist/include" /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/_virtualenv/bin/python /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/../config/nsinstall.py -m 644 "../config/nsStaticComponents.h" "../dist/include" rm -f ../config/final-link-comps ../config/final-link-libs ../config/final-link-comp-names /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/_virtualenv/bin/python ../toolkit/xre/make-platformini.py --print-buildid > buildid /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/_virtualenv/bin/python ../config/Preprocessor.py -DMOZ_GLUE_IN_PROGRAM -DNO_NSPR_10_SUPPORT -DJSGC_INCREMENTAL=1 -DMOZ_ENABLE_PROFILER_SPS=1 -DMOZILLA_VERSION=\"22.0a1\" -DMOZILLA_VERSION_U=22.0a1 -DMOZILLA_UAVERSION=\"22.0\" -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_UNAME_DOMAINNAME_FIELD=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_SYS_BITYPES_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_GNU_LIBC_VERSION_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_MALLOC_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_CPUID_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_SYSMACROS_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_NETINET_IN_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLOPEN=1 -DHAVE_DLADDR=1 -DHAVE_MEMMEM=1 -DFUNCPROTO=15 -D_REENTRANT=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_TRUNCATE64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_RES_NINIT=1 -DHAVE_GNU_GET_LIBC_VERSION=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DMALLOC_H=\<malloc.h\> -DHAVE_STRNDUP=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_MEMALIGN=1 -DHAVE_VALLOC=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_X11=1 -DMOZ_WIDGET_GTK2=1 -DMOZ_WIDGET_GTK=2 -DMOZ_PDF_PRINTING=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DMOZ_PANGO=1 -DMOZ_ENABLE_GIO=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_DBUS=1 -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_PEERCONNECTION=1 -DMOZ_SCTP=1 -DMOZ_SRTP=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_RAW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DMOZ_DASH=1 -DMOZ_MEDIA_NAVIGATOR=1 -DMOZ_VP8=1 -DMOZ_VP8_ERROR_CONCEALMENT=1 -DMOZ_VP8_ENCODER=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_SPEEX_RESAMPLER=1 -DMOZ_SOUNDTOUCH=1 -DMOZ_CUBEB=1 -DMOZ_MEDIA=1 -DMOZ_VORBIS=1 -DMOZ_OPUS=1 -DMOZ_WEBVTT=1 -DMOZ_CUBEB=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X86_ASM=1 -DMOZ_FLEXBOX=1 -DMOZ_WEBAPP_RUNTIME=1 -DMOZ_SIGNING=1 -DMOZ_UPDATER=1 -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DGL_PROVIDER_GLX=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DSIZEOF_INT_P=4 -DMOZ_MEMORY_SIZEOF_PTR_2POW=2 -DMOZ_MEMORY=1 -DMOZ_MEMORY_LINUX=1 -DJSGC_INCREMENTAL=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_FT_SELECT_SIZE=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DMOZ_TREE_PIXMAN=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_COMMON=1 -DMOZ_SERVICES_CRYPTO=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SERVICES_METRICS=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightly -DMOZ_B2G_VERSION=\"1.0.0\" -DMOZ_B2G_OS_NAME=\"\" -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"22.0a1\" -DFIREFOX_VERSION=22.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_TELEMETRY_DISPLAY_REV=2 -DMOZ_DATA_REPORTING=1 -DMOZ_DLL_SUFFIX=\".so\" -DHAVE_POSIX_FALLOCATE=1 -DXP_UNIX=1 -DMOZ_ACCESSIBILITY_ATK=1 -DATK_MAJOR_VERSION=1 -DATK_MINOR_VERSION=28 -DATK_REV_VERSION=0 \ -DMOZ_TREE_CAIRO=1 \ -DMOZ_TREE_PIXMAN=1 \ -DMOZ_NATIVE_HUNSPELL= \ -DMOZ_NATIVE_BZ2= \ -DMOZ_NATIVE_ZLIB= \ -DMOZ_NATIVE_PNG= \ -DMOZ_NATIVE_JPEG= \ -DMOZ_NATIVE_LIBEVENT= \ -DMOZ_NATIVE_LIBVPX= \ ../config/system-headers | /usr/bin/perl ../nsprpub/config/make-system-wrappers.pl system_wrappers /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/nsinstall -R system_wrappers ../dist /builds/slave/m-cen-lx-l10n-ntly-00000000000/build/mozilla-central/config/nsinstall: cannot access system_wrappers: No such file or directory make: *** [export] Error 1 No idea what that is, maybe someone from releng or Mike know?
Older x86-64 builds are getting update offers to 20130305: https://aus3.mozilla.org/update/3/Firefox/22.0a1/20130228100310/Linux_x86_64-gcc3/fr/nightly/Linux%203.5.0-23-generic%20%28GTK%202.24.13%29/default/default/update.xml?force=1
Summary: No Nightly updates for localized builds on Linux64 → No Nightly updates anymore for localized builds on Linux
Looks to be a regression. The first time it appeared was on March 01. We haven't had those failures before. http://mozmill-ci.blargon7.com/#/update/report/66aad0ebfa7a01580628b3af4c9bbca7
Keywords: regression
There are two problems here: linux32 repacks are failing with the error Axel mentioned linux64 repacks today downloaded the wrong en-US build, and so didn't generate an update to todays' build Let's try and keep these issues separate
Right. I would say that Axel should file a new bug then. The issues we have seen with the updates is not related to the build failure and apply to both x86 and x86-64.
(In reply to Henrik Skupin (:whimboo) from comment #3) > Same issue for the x86 build on Linux from 20130304. We do not offer an > update to this build if you have the build from 20130303 installed: > > https://aus3.mozilla.org/update/3/Firefox/22.0a1/20130303030835/Linux_x86- > gcc3/fr/nightly/Linux%203.2.0-37-generic-pae%20%28GTK%202.24.10%29/default/ > default/update.xml?force=1 > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-04-03- > 09-33-mozilla-central-l10n/firefox-22.0a1.fr.linux-x86_64.tar.bz2 Let me correct that please. The build I wanted to refer here is: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-03-03-08-35-mozilla-central-l10n/firefox-22.0a1.fr.linux-i686.tar.bz2 FTP showed me March 4th as the time it has been uploaded. But that was in my timezone. So the snippet above can indeed not find an update. Which makes that case a dupe of bug 808550. Looks like there are really only 64bit builds which are affected here.
Hardware: All → x86_64
Summary: No Nightly updates anymore for localized builds on Linux → No Nightly updates anymore for localized builds on Linux 64
Unghost filed bug 848452 on the linux32 build bustage.
Depends on: 848584
Happened again last Friday most likely due to multiple respins of Aurora during that day. This is somewhat blocking us from signing off for correct build updates. Especially when a failure like this will hit us for rapid betas, we will be in trouble. Would be great to see it fixed.
Whiteboard: [automation-blocked]
Summary: No Nightly updates anymore for localized builds on Linux 64 → Missing Nightly updates for localized builds on Linux 64 if respins of daily builds happen
(In reply to Henrik Skupin (:whimboo) from comment #10) > This is somewhat blocking us from signing off for correct build updates. > Especially when a failure like this will hit us for rapid betas, we will be > in trouble. Would be great to see it fixed. This is an aside, but I do want to note that depending on how rapid betas are implemented that this is not necessarily the case.
This happened again now for it Aurora builds on OS X. The pulse message didn't contain the previous_buildid.
Sorry, the last comment was intended for a different bug.
Whiteboard: [automation-blocked] → [qa-automation-blocked]
Duping against bug 858953 because it seems to contain more information about how to fix the issue.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Whiteboard: [qa-automation-blocked]
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.