Closed
Bug 1192308
Opened 9 years ago
Closed 9 years ago
uploads to new s3 buckets are significantly slower than uploading to stage
Categories
(Cloud Services :: General, defect)
Cloud Services
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bhearsum, Assigned: oremj)
References
Details
An upload via the new hosts took about 3.5min: 16:06:46 INFO - '../../dist/firefox-42.0a1.en-US.linux-i686.tar.bz2' '../../dist/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi' '../../dist/firefox-42.0a1.en-US.linux-i686.common.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip' '../../dist//firefox-42.0a1.en-US.linux-i686.txt' '../../dist//firefox-42.0a1.en-US.linux-i686.json' '../../dist//firefox-42.0a1.en-US.linux-i686.mozinfo.json' '../../dist//test_packages.json' '../../dist/jsshell-linux-i686.zip' ../../dist/host/bin/mar ../../dist/host/bin/mbsdiff \ 16:06:46 INFO - '../../dist//firefox-42.0a1.en-US.linux-i686.checksums' '../../dist//firefox-42.0a1.en-US.linux-i686.checksums'.asc 16:09:55 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.tar.bz2 16:09:55 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.langpack.xpi 16:09:56 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.common.tests.zip 16:10:00 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip 16:10:01 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip 16:10:04 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip 16:10:05 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip 16:10:07 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip 16:10:08 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip 16:10:08 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.txt 16:10:08 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.json 16:10:08 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.mozinfo.json 16:10:08 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/test_packages.json 16:10:09 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/jsshell-linux-i686.zip 16:10:09 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/mar 16:10:09 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/mbsdiff 16:10:09 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.checksums 16:10:09 INFO - http://releases.mozilla.org/pub/firefox/tinderbox-builds/ash-linux/1438813360/firefox-42.0a1.en-US.linux-i686.checksums.asc 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.tar.bz2 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.common.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.txt 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.json 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.mozinfo.json 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/test_packages.json 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/jsshell-linux-i686.zip 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/host/bin/mar 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/host/bin/mbsdiff 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.checksums 16:10:11 INFO - Uploading /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.checksums.asc 16:10:11 INFO - Running post-upload command: post_upload.py --tinderbox-builds-dir ash-linux -p firefox -i 20150805152240 --revision 41e0d9f4421bbdcf86ef85fef434ccbbb31eb1d1 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds --bucket-prefix net-mozaws-stage-delivery 16:10:11 INFO - Upload complete Whereas a similar job uploading to ftp/stage took under a minute: 08:46:24 INFO - /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/m-cen-lx-000000000000000000000/build/src/build/upload.py --base-path ../../dist \ 08:46:24 INFO - '../../dist/firefox-42.0a1.en-US.linux-i686.tar.bz2' '../../dist/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi' '../../dist/firefox-42.0a1.en-US.linux-i686.common.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip' '../../dist//firefox-42.0a1.en-US.linux-i686.txt' '../../dist//firefox-42.0a1.en-US.linux-i686.json' '../../dist//firefox-42.0a1.en-US.linux-i686.mozinfo.json' '../../dist//test_packages.json' '../../dist/jsshell-linux-i686.zip' ../../dist/host/bin/mar ../../dist/host/bin/mbsdiff \ 08:46:24 INFO - '../../dist//firefox-42.0a1.en-US.linux-i686.checksums' '../../dist//firefox-42.0a1.en-US.linux-i686.checksums'.asc 08:47:04 INFO - sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-linux', '-p', 'firefox', '-i', '20150807041443', '--revision', '3e51753a099f8014b3dc37ed3fa27b887842735b', '--release-to-tinderbox-dated-builds', '--release-to-latest-tinderbox-builds', '/tmp/tmp.nnMJL6uXPv/', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.tar.bz2', '/tmp/tmp.nnMJL6uXPv/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.common.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.reftest.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.txt', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.json', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.mozinfo.json', '/tmp/tmp.nnMJL6uXPv//test_packages.json', '/tmp/tmp.nnMJL6uXPv//jsshell-linux-i686.zip', '/tmp/tmp.nnMJL6uXPv/host/bin/mar', '/tmp/tmp.nnMJL6uXPv/host/bin/mbsdiff', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.checksums', '/tmp/tmp.nnMJL6uXPv//firefox-42.0a1.en-US.linux-i686.checksums.asc'] 08:47:05 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.tar.bz2 08:47:05 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.langpack.xpi 08:47:05 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.common.tests.zip 08:47:05 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip 08:47:05 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.txt 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.json 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.mozinfo.json 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/test_packages.json 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/jsshell-linux-i686.zip 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/mar 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/mbsdiff 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.checksums 08:47:06 INFO - http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1438946083/firefox-42.0a1.en-US.linux-i686.checksums.asc 08:47:06 INFO - ln -sfnv 1438946083 /home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux/latest 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.tar.bz2 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.common.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.txt 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.json 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.mozinfo.json 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/test_packages.json 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/jsshell-linux-i686.zip 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/host/bin/mar 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/host/bin/mbsdiff 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.checksums 08:47:07 INFO - Uploading /builds/slave/m-cen-lx-000000000000000000000/build/src/obj-firefox/dist/firefox-42.0a1.en-US.linux-i686.checksums.asc 08:47:07 INFO - Running post-upload command: post_upload.py --tinderbox-builds-dir mozilla-central-linux -p firefox -i 20150807041443 --revision 3e51753a099f8014b3dc37ed3fa27b887842735b --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds 08:47:07 INFO - Upload complete This might be totally expected and acceptable, but since it's such a huge slow down (roughly 4x), I wanted to call it out explicitly. It wouldn't surprise me if this gets even slower when the new upload hosts are under full load (like ftp/stage currently is), depending where the bottleneck lies.
|
Reporter | |
Comment 1•9 years ago
|
||
Jeremy, assigning to you because it seems like the right way to get your attention.
Assignee: nobody → oremj
|
Assignee | |
Comment 2•9 years ago
|
||
It's expected that this will be slower. The previous scenario was upload over ssh on local network. Now we have upload over internet via ssh and then from that host to s3, over the internet again.
|
||
Comment 3•9 years ago
|
||
Are the uploads sequential, or is there some parallelism ?
|
|
Assignee | |
Comment 4•9 years ago
|
||
The uploads to s3, by post_upload, are sequential, but I do plan on adding parallel uploads at some point.
|
|
||
Comment 5•9 years ago
|
||
I tweaked our uploading so that we get logging with legit timestamps and ordering (via unbuffered python output), and the resulting push had these upload times job start post_upload complete scp upload post_upload ~ size (MB) scp MB/s linux32 opt 05:22:39 05:24:44 05:25:04 00:02:05 00:00:20 330 2.6 linux64 asan debug 05:22:44 05:25:45 05:26:29 00:03:01 00:00:44 730 4.0 osx opt 06:23:29 06:29:26 06:29:55 00:05:57 00:00:29 470 1.3 osx debug 05:06:20 05:10:14 05:10:37 00:03:54 00:00:23 280 1.2 win32 – no useful log So most of the time is in the scp rather the new post_upload. Surprisingly linux is fastest, despite a AWS -> SCL3 -> AWS network path, while mac is fairly slow on just SCL3 -> AWS. There may be some variation in speeds, and this level of data isn't sufficient.
|
|
||
Comment 6•9 years ago
|
||
We've redone the routing in bug 1202513, so AWS slaves should spend less time in scp upload. Needs verification.
|
|
Assignee | |
Comment 7•9 years ago
|
||
Going to close this as fixed. Please reopen if anything else is needed.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•