1102489 - m-i and fx-team test logs are being uploaded to ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jordan Lund (:jlund)]

example: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/2014-11-18-21-47-12-mozilla-inbound/

these should not be uploaded there. trying to debug right now

working backwards, this is the logic I see:

we call post_upload.py to upload logs:
http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#116

in order to upload to nightly/firefox/$date-branch, we need --release-dated:
http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#180

in postrun.py case, that is dictated by log_uploader.py:
http://mxr.mozilla.org/build/source/buildbotcustom/bin/log_uploader.py#310

which is defined in postrun.py here:
http://mxr.mozilla.org/build/source/buildbotcustom/bin/postrun.py#65

builder.name comes from FakeBuilder which is basically just the dirname to the builder pickle:
http://mxr.mozilla.org/build/source/buildbotcustom/bin/postrun.py#148

so in the following example log output from a postrun.py call:
[snippet from a linux test master /dev/shm/queue/commands/logs/{log}]
Running [u'/builds/buildbot/tests1-linux/bin/python', u'/builds/buildbot/tests1-linux/lib/python2.7/site-packages/buildbotcustom/bin/postrun.py', u'-c', u'/builds/buildbot/te    sts1-linux/master/postrun.cfg', u'--master-name', u'buildbot-master103.srv.releng.scl3.mozilla.com:/builds/buildbot/tests1-linux/master', u'--master-incarnation', u'pid24834-    boot1414971241', u'/builds/buildbot/tests1-linux/master/mozilla-inbound_ubuntu32_hw_test-dromaeojs/4194', u'55641613']
2014-11-20 12:40:27,780 - Loading build pickle
2014-11-20 12:40:27,964 - Build info: {'platform': u'linux', 'product': 'firefox', 'branch': 'mozilla-inbound'}
2014-11-20 12:40:27,964 - uploading log
2014-11-20 12:40:27,964 - Running ['/builds/buildbot/tests1-linux/bin/python', '/builds/buildbot/tests1-linux/lib/python2.7/site-packages/buildbotcustom/bin/log_uploader.py',     '-r', '2', '-t', '10', '--master-name', u'bm103-tests1-linux', '--product', 'firefox', '--platform', u'linux', '--branch', 'mozilla-inbound', '--user', u'ffxbld', '-i', u'/h    ome/cltbld/.ssh/ffxbld_dsa', u'stage.mozilla.org', '/builds/buildbot/tests1-linux/master/mozilla-inbound_ubuntu32_hw_test-dromaeojs', '4194']
2014-11-20 12:40:27,964 - command: /builds/buildbot/tests1-linux/bin/python /builds/buildbot/tests1-linux/lib/python2.7/site-packages/buildbotcustom/bin/log_uploader.py -r 2     -t 10 --master-name bm103-tests1-linux --product firefox --platform linux --branch mozilla-inbound --user ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org /builds/bui    ldbot/tests1-linux/master/mozilla-inbound_ubuntu32_hw_test-dromaeojs 4194
2014-11-20 12:40:28,866 - Running post_upload.py --tinderbox-builds-dir mozilla-inbound-linux -b mozilla-inbound -p firefox -i 20141120114905 --revision 7b95b87d4085e4454ca14    6f4e8f3f47b8c4b1389 --release-to-tinderbox-dated-builds --release-to-dated /tmp/tmp.HlRigrxNmS /tmp/tmp.HlRigrxNmS/mozilla-inbound_ubuntu32_hw_test-dromaeojs-bm103-tests1-lin    ux-build4194.txt.gz
sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-inbound-linux', '-b', 'mozilla-inbound', '-p', 'firefox', '-i', '20141120114905', '--revision',     '7b95b87d4085e4454ca146f4e8f3f47b8c4b1389', '--release-to-tinderbox-dated-builds', '--release-to-dated', '/tmp/tmp.HlRigrxNmS', '/tmp/tmp.HlRigrxNmS/mozilla-inbound_ubuntu32    _hw_test-dromaeojs-bm103-tests1-linux-build4194.txt.gz']
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/2014-11-20-11-49-05-mozilla-inbound/mozilla-inbound_ubuntu32_hw_test-dromaeojs-bm103-tests1-linux-build4194.txt    .gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1416512945/mozilla-inbound_ubuntu32_hw_test-dromaeojs-bm103-tests1-linux-build4194.txt.gz

/builds/buildbot/tests1-linux/master/mozilla-inbound_ubuntu32_hw_test-dromaeojs/4194 seems to be the build_path:
http://mxr.mozilla.org/build/source/buildbotcustom/bin/postrun.py#360

which means that builder.name is mozilla-inbound_ubuntu32_hw_test-dromaeojs (and the pickle is the file 4194)

but you can see that 'nightly' is not in mozilla-inbound_ubuntu32_hw_test-dromaeojs so I have no idea how in the above log snippet we are uploading to:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/2014-11-20-11-49-05-mozilla-inbound/mozilla-inbound_ubuntu32_hw_test-dromaeojs-bm103-tests1-linux-build4194.txt.gz

Comment 1

[Jordan Lund (:jlund)]

it turns out that pgo_build can be given True and False and behave as expected...

our old logic: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2589

but nightly_build is different, we look for the presence of its existence so the True/False part is irrelevant. old logic: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2586

see how http://mxr.mozilla.org/build/source/buildbotcustom/bin/log_uploader.py#105 doesn't check for True

Comment 2

[Jordan Lund (:jlund)]

Attachment: 141120_dont_pass_nightly_build_prop_to_test_jobs.patch

see https://bugzilla.mozilla.org/show_bug.cgi?id=1102489#c1 for explanation

Comment 3

[Jordan Lund (:jlund)]

Attachment: 141120_dont_pass_nightly_build_prop_to_test_jobs2.patch

new and improved efficiency!

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8526321 [details] [diff] [review]
141120_dont_pass_nightly_build_prop_to_test_jobs2.patch

It looks like http://mxr.mozilla.org/build/source/buildbotcustom/bin/log_uploader.py#105 is returning u'False' which is evaluating to a True.

I'd still like to use this patch even if it's not perfect as it will fix the bug right now.

I'd imagine this is why we didn't set nightly_build:False like we do with pgo_build in the old buildbot way.

Fixes could include giving {nightly_build: ''} in sendchange or checking if nightly_build == 'False' but those both seem far hackier than this patch.

The best fix will require investigating whether it is buildbot serializing the sendchange props incorrectly or if cPickle is deserializing incorrectly. And then either patch one of those or figure out acceptable inputs for serializing.

I've wasted enough time today trying to fix this so I'd prefer to move on to my current goals.

Comment 5

[Jordan Lund (:jlund)]

see comment 4 and request to proceed with my current patch

Comment 6

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/2453fb0cf052

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8526321 [details] [diff] [review]
141120_dont_pass_nightly_build_prop_to_test_jobs2.patch

Review of attachment 8526321 [details] [diff] [review]:
-----------------------------------------------------------------

this was r+'ed already over email