Closed Bug 887678 Opened 11 years ago Closed 11 years ago

linux64 static analysis builds upload to latest-mozilla-central

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: AndreeaMatei, Unassigned)

References

Details

Attachments

(3 files, 1 obsolete file)

application.ini from linux64
599 bytes, text/plain
Details
Build log
638.37 KB, text/plain
Details
disable other uploads
524 bytes, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review 
Our mozmill test for crashreporter has failed a few times on Linux due to the fact that sometimes the crashreporter is not enabled inside the application.ini file of the build (bug 880220).

Here I found the 'it' build which shows this issue:
ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla-central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2

We also have seen it in May 31st, June 6th and June 22th, but for the older ones there are no l10n builds on ftp anymore.

It never happened with en-US, only it, fr, de (these are the one we test so far, could be other locales as well).
Blocks: 880220
Ted and Ben, do you have any idea why it's not always enabled in localized builds?
Version: 25 Branch → 24 Branch
This would require me to know way more about the l10n process than I currently do. I can tell you that application.ini is informational only nowadays, its contents are compiled into the browser binary directly, so you should sanity check that crash reporting is actually enabled in these builds (you can use my Crash Me Now extension: http://people.mozilla.com/~tmielczarek/crashme/ )
If there's actually a problem here, it's probably a question for Axel.
Flags: needinfo?(community)
That's not something I can help with.

I suspect that for some or all l10n repacks, MOZILLA_OFFICIAL doesn't match with what en-US is built with. That is, if the repacks actually rebuild the application.ini, which would be packaging code I don't understand.

Reading Ted's comment, that probably doesn't have an impact on whether crash reporting actually works or not, though.
Flags: needinfo?(community)
I found the difference from enabled or not in that application.ini file, but it is certainly disabled for those builds where our test has failed (see the link in first comment as example). We check that using nsICrashReporter:
http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/functional/testInstallation/testBreakpadInstalled.js#l45

Shouldn't the crashreporter be enabled on all builds by default? Even localized? I need this information to know how to further proceed with our test, if we need to update the way we handle it.
Yes, it definitely should be.
(In reply to Andreea Matei [:AndreeaMatei] from comment #0)
> Here I found the 'it' build which shows this issue:
> ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla-
> central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2

Ben, do we have the repack logs still available for that l10n build? If yes, is there something suspicious in there? This bug occurs rather rarely, so I could imaging something is broken with the builder, and it incorrectly writes out the application.ini file.
(In reply to Henrik Skupin (:whimboo) [away 06/28 - 07/07] from comment #7)
> (In reply to Andreea Matei [:AndreeaMatei] from comment #0)
> > Here I found the 'it' build which shows this issue:
> > ftp://ftp.mozilla.org/pub/firefox/nightly/2013-06-22-03-10-42-mozilla-
> > central-l10n/firefox-24.0a1.it.linux-x86_64.tar.bz2
> 
> Ben, do we have the repack logs still available for that l10n build? If yes,
> is there something suspicious in there? This bug occurs rather rarely, so I
> could imaging something is broken with the builder, and it incorrectly
> writes out the application.ini file.

No. Looks like we only keep a weeks worth of l10n builds/logs.
Ok, so when it happens again, we will make sure to reply immediately. That will hopefully help us to nail down the reported issue.
I can reproduce again with today's it, de and fr builds from here:
ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/
Setting needinfo flag for Ben so we do not miss to check for the logs of those builds until they will get deleted.
Flags: needinfo?(bhearsum)
I checked all 4 files currently available for it, only linux64 is missing the line "Enabled = 1" in CrashReporter section.
I wonder if 23.0 builds are also affected. But for now asking blocking for 24.0 and 25.0.
status-firefox23: --- → ?
status-firefox24: --- → affected
status-firefox25: --- → affected
tracking-firefox24: --- → ?
tracking-firefox25: --- → ?
Attached file Build log — 
If I'm not mistaken, based on the buildid in application.ini this should be the build log.
(In reply to Andreea Matei [:AndreeaMatei] from comment #10)
> I can reproduce again with today's it, de and fr builds from here:
> ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/

Logs are here: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/

Grab 'em while you can.
Flags: needinfo?(bhearsum)
I rechecked and the request came through the job #495. So the correct build log has been already attached by flod. Ben, can you please have a look at and check if something is suspicious?
(In reply to Henrik Skupin (:whimboo) from comment #16)
> I rechecked and the request came through the job #495. So the correct build
> log has been already attached by flod. Ben, can you please have a look at
> and check if something is suspicious?

I don't have anymore insight into this than you do -- none of the build steps failed, the job was green. I know only a little bit of the l10n and packaging parts of the build system, and nothing in the logs stands out to me.
Interestingly, the build id and sourcestamp are different between the it 64 build, the it 32 build, and en-US 32/64. I get three different ones, actually.

64 it:

fx_revision 5976b9c673f8
buildid 20130715211256


32 it:

fx_revision 5e191a26d909
buildid 20130715100109

en-US 32 and 64:

[App]
Vendor=Mozilla
Name=Firefox
Version=25.0a1
BuildID=20130715030202
SourceRepository=http://hg.mozilla.org/mozilla-central
SourceStamp=ff0a372e3170
ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}


Ben, any idea?
It might not be related to the number of builds per day, given that the dates from the original comment do not have multiple nightly builds per day. So it should be something different.
(In reply to Ben Hearsum [:bhearsum] from comment #19)
> Possibly of note, there were three nightlies yesterday:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-03-02-02-
> mozilla-central-l10n/
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-10-01-09-
> mozilla-central-l10n/
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-
> mozilla-central-l10n/

Ah probably grabbed random stuff then. Which was triggered by the fact that there aren't 32bit Italian builds on https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/

More dark to stab in.
I know bug 858953 is for updates only but I wonder if that also has an effect in how we rebuild for all of our locales.
It looks like some of the l10n builds are repackaged from debug builds and others are not. The de build is a debug build, while sv-SE is an opt build.

For some reason the en-US build in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ is a debug build at the time that some l10n builders fetch the nightly.
The ASAN build is in that directory too, maybe it's grabbing that?
(In reply to Hasse from comment #23)
> It looks like some of the l10n builds are repackaged from debug builds and
> others are not. The de build is a debug build, while sv-SE is an opt build.
> 
> For some reason the en-US build in
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> central/ is a debug build at the time that some l10n builders fetch the
> nightly.

Can you point me at a specific repack that was based on a debug build?
Flags: needinfo?(hasse)
The sv-SE build is repackaged from this build:
2013-07-16 03:26:54 URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [34679000/34679000] -> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1]

And de is repackaged from this build:
2013-07-16 03:27:53 URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [33885535/33885535] -> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1]

Notice the different file size. Could it be that a debug build is uploaded into this directory sometime after 2013-07-16 03:26:54?

The build log for the de build (debug):
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-de-bm61-build1-build682.txt.gz

The sv-SE build log (opt):
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-sv-SE-bm64-build1-build562.txt.gz
Flags: needinfo?(hasse)
(In reply to Hasse from comment #27)
> The sv-SE build is repackaged from this build:
> 2013-07-16 03:26:54
> URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [34679000/34679000] ->
> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1]

This matches the filesize of https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-07-15-21-12-56-mozilla-central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2

> And de is repackaged from this build:
> 2013-07-16 03:27:53
> URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> central/firefox-25.0a1.en-US.linux-x86_64.tar.bz2 [33885535/33885535] ->
> "firefox-25.0a1.en-US.linux-x86_64.tar.bz2" [1]

And this is https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1373947976/firefox-25.0a1.en-US.linux-x86_64.tar.bz2

And indeed, in that log I see:
Running post-upload command: post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i 20130715211256 --revision 5976b9c673f8 --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated

The "release-to-latest" part is causing the builds to upload to https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/, which is causing this problem.

This is actually a really big issue, because it means that anyone who downloads "Nightly" in this window will end up with a static analysis build that never updates.
Interesting, Dave Hunt has seen the same some weeks ago when trying to install a nightly through http://nightly.mozilla.org/. He also got a debug build served! I think that's somewhat really problematic and would need investigation. We shouldn't offer debug builds to our nightly users.
Summary: Some localized builds don't have crashreporter enabled in application.ini → linux64 static analysis builds upload to latest-mozilla-central
Ben, which builds from which branches would that affect? Only nightly or also aurora?
Severity: normal → major
--- /home/bhearsum/tmp/a.txt	2013-07-16 11:41:46.041622353 -0400
+++ /home/bhearsum/tmp/b.txt	2013-07-16 11:40:21.653621016 -0400
@@ -15288,7 +15288,7 @@
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
-    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ionmonkey-linux64-dbg-st-an -b ionmonkey -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ionmonkey-linux64-dbg-st-an -b ionmonkey -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
     C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ionmonkey',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {}
     DisconnectStep {'alwaysRun': True, 'command': ['python',  'tools/buildfarm/maintenance/count_and_reboot.py',  '-f',  '../reboot_count.txt',  '-n',  '1',  '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}
@@ -17439,7 +17439,7 @@
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
-    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir mozilla-central-linux64-dbg-st-an -b mozilla-central -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
     C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {}
     DisconnectStep {'alwaysRun': True, 'command': ['python',  'tools/buildfarm/maintenance/count_and_reboot.py',  '-f',  '../reboot_count.txt',  '-n',  '1',  '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}
@@ -18532,7 +18532,7 @@
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
-    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-oak',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir oak-linux64-dbg-st-an -b oak -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-oak',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir oak-linux64-dbg-st-an -b oak -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
     C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-oak',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-oak',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {}
     DisconnectStep {'alwaysRun': True, 'command': ['python',  'tools/buildfarm/maintenance/count_and_reboot.py',  '-f',  '../reboot_count.txt',  '-n',  '1',  '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}
@@ -19426,7 +19426,7 @@
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'BuildID'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_build_id', 'property': 'buildid', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Version'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_version', 'property': 'appVersion', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['python',  'build/config/printconfigsetting.py',  'build/obj-firefox/dist/bin/application.ini',  'App',  'Name'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_app_name', 'property': 'appName', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
-    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ux',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ux-linux64-dbg-st-an -b ux -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    C {'command': ['make', 'upload'], 'description': ['make', 'upload'], 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ux',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'POST_UPLOAD_CMD': <WithProperties "post_upload.py --tinderbox-builds-dir ux-linux64-dbg-st-an -b ux -p firefox -i %(buildid)s --revision %(got_revision)s --release-to-tinderbox-dated-builds --release-to-dated">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'UPLOAD_HOST': 'stage.mozilla.org',  'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_dsa',  'UPLOAD_TO_TEMP': '1',  'UPLOAD_USER': 'ffxbld'}, 'extract_fn': <function parse_make_upload>, 'haltOnFailure': True, 'locks': [<buildbot.locks.LockAccess>], 'log_eval_func': <function <lambda>>, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_upload', 'property': None, 'retry': True, 'strip': True, 'target': 'mozilla-centos6-x86_64', 'timeout': 2460, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
     C {'command': ['rm', '-rf', 'build'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ux',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'log_eval_func': None, 'logfiles': {}, 'name': 'rm_builddir', 'timeout': 5400, 'usePTY': 'slave-config', 'workdir': '.'} {}
     C {'command': ['ccache', '-s'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_BASEDIR': <WithProperties "%(basedir:-)s">,  'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'IS_NIGHTLY': 'yes',  'LC_ALL': 'C',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SYMBOLS_EXTRA_BUILDID': 'linux64-dbg-st-an',  'MOZ_UPDATE_CHANNEL': 'nightly-ux',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_dsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1'}, 'flunkOnFailure': False, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'print_ccache_stats', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': None} {}
     DisconnectStep {'alwaysRun': True, 'command': ['python',  'tools/buildfarm/maintenance/count_and_reboot.py',  '-f',  '../reboot_count.txt',  '-n',  '1',  '-z'], 'description': ['maybe rebooting'], 'descriptionDone': None, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'maybe_rebooting', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}
Attachment #776430 - Flags: review?(rail)
Any branch that runs static analysis builds. Looks like that's most mozilla-central based branches.

My fix will be in production today, though, so this shouldn't be an issue starting with the next set of nightlies.
Severity: major → critical
Attachment #776430 - Flags: review?(rail) → review+
status-firefox23: ? → ---
status-firefox24: affected → ---
status-firefox25: affected → ---
tracking-firefox24: ? → ---
tracking-firefox25: ? → ---
Component: Breakpad Integration → Release Engineering: Automation (General)
Product: Toolkit → mozilla.org
QA Contact: catlee
Version: 24 Branch → other
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
I did some more grepping and found more debug-ey builds that are doing this, too.
Attachment #776430 - Attachment is obsolete: true
Attachment #776474 - Flags: review?(rail)
Attachment #776474 - Flags: review?(rail) → review+
Comment on attachment 776474 [details] [diff] [review]
disable other uploads

in production
Attachment #776474 - Flags: checked-in+
Shouldn't be a problem starting with the next set of nightlies.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
I have to thank everyone on this bug who was involved in getting this critical issue fixed. I never thought that with our mozmill test failure such an important underlying issue would open. It's great to see that team work!
Product: mozilla.org → Release Engineering
While doing regression tests for a bug, which I found in Nightly 26.0a1, I also tested the following build:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.linux-x86_64.tar.bz2

As it turns out it is a debug build again. So the underlying issue here has not been fully fixed.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
When you check the folder itself you will notice that this linux x86_64 build has been uploaded again two days later:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/

All builds are from Aug 19th but the linux 64bit one is from Aug 21st:

The json file says:

 "buildid": "20130819030205", 
  "cc": "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang", 
  "cxx": "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang++", 

So it's really a static analysis build again.
I have checked all the September builds of Nightly on ftp.mo but wasn't able to find any other instance of this problem. Looks like it's still happening very rarely.
Status: REOPENED → ASSIGNED
(In reply to Henrik Skupin (:whimboo) from comment #39)
> I have checked all the September builds of Nightly on ftp.mo but wasn't able
> to find any other instance of this problem. Looks like it's still happening
> very rarely.

In the future, can you please file new bugs for issues that recur after so long? This almost got lost because it was already assigned to me.

We added a new static analysis build type in bug 910919, but I don't see it running anywhere but try, so I'm not convinced that's the issue. I look into this more later/tomorrow.
Assignee: bhearsum → nobody
(In reply to Henrik Skupin (:whimboo) from comment #38)
> When you check the folder itself you will notice that this linux x86_64
> build has been uploaded again two days later:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-
> 02-05-mozilla-central/
> 
> All builds are from Aug 19th but the linux 64bit one is from Aug 21st:
> 
> The json file says:
> 
>  "buildid": "20130819030205", 
>   "cc":
> "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang", 
>   "cxx":
> "/builds/slave/m-cen-l64-dbg-st-an-ntly-00000/build/clang/bin/clang++", 
> 
> So it's really a static analysis build again.

An upload from a recent dbg-st-an build doesn't make it look like these builds are uploading to latest-mozilla-central:
sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-linux64-dbg-st-an', '-p', 'firefox', '-i', '20130826171944', '--revision', 'c0d1baa50a64', '--release-to-tinderbox-dated-builds', '/tmp/tmp.PIhFcIablt/', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.tar.bz2', '/tmp/tmp.PIhFcIablt/linux-x86_64/xpi/firefox-26.0a1.en-US.langpack.xpi', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.txt', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.json', '/tmp/tmp.PIhFcIablt/host/bin/mar', '/tmp/tmp.PIhFcIablt/host/bin/mbsdiff', '/tmp/tmp.PIhFcIablt//firefox-26.0a1.en-US.linux-x86_64.checksums']
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1377562784/firefox-26.0a1.en-US.linux-x86_64.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-dbg-st-an/1377562784/firefox-26.0a1.en-US.langpack.xpi
The log from the build you linked to tells a different story:
sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-linux64-dbg-st-an', '-b', 'mozilla-central', '-p', 'firefox', '-i', '20130819030205', '--revision', 'c8c9bd74cc40', '--release-to-tinderbox-dated-builds', '--release-to-dated', '/tmp/tmp.LrSlcDIeqe/', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.tar.bz2', '/tmp/tmp.LrSlcDIeqe/linux-x86_64/xpi/firefox-26.0a1.en-US.langpack.xpi', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.txt', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.json', '/tmp/tmp.LrSlcDIeqe/host/bin/mar', '/tmp/tmp.LrSlcDIeqe/host/bin/mbsdiff', '/tmp/tmp.LrSlcDIeqe//firefox-26.0a1.en-US.linux-x86_64.checksums']
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.linux-x86_64.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08/2013-08-19-03-02-05-mozilla-central/firefox-26.0a1.en-US.langpack.xpi

Note that "-b mozilla-central" and "--release-to-dated" are present here, but not in the more recent post_upload.py command I pasted in my previous comment. Still digging into this.
Looks like something happened around August 11th or 12th. This grepping shows that "release-to-dated" only started showing up on the 12th:
-bash-4.1$ for i in `find . -iname "*-mozilla-central"`; do zgrep -L release-to-dated $i/mozilla-central-linux64-dbg-st-an-nightly*.gz; done
./2013-08-01-03-02-23-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm66-build1-build24.txt.gz
...
./2013-08-11-03-02-25-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm62-build1-build2.txt.gz
-bash-4.1$ for i in `find . -iname "*-mozilla-central"`; do zgrep -l release-to-dated $i/mozilla-central-linux64-dbg-st-an-nightly*.gz; done
./2013-08-12-03-02-09-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly-bm58-build1-build26.txt.gz

And then around the 31st of August these files start appearing altogether:
gzip: ./2013-08-31-03-02-24-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory
gzip: ./2013-08-31-03-17-59-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory
gzip: ./2013-08-31-03-14-59-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory
...
gzip: ./2013-09-29-03-02-02-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory
gzip: ./2013-09-30-03-02-05-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory
gzip: ./2013-09-30-07-12-02-mozilla-central/mozilla-central-linux64-dbg-st-an-nightly*.gz: No such file or directory

Given that, I don't think this is an issue anymore...I'd still like to get to the bottom of it though.
I'm still baffled by this, but I don't have any more time to investigate it right now. I really do think it's fixed again based on the logs from the past month, so I'm closing this. Please file a new bug and reference this one if we hit it again.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: