1186297 - Switch ash branch to upload via new S3 frontends

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

... as an initial loadtest. Ash was suggested.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [ash] Stop using stage.mozilla.org

I've merged m-c to ash again, and this patch should take care of all the desktop, android, and the public parts of device builds. The exception is b2g manifests, which will be broken by a mismatched key and host. I am unsure if we still need these, but if we do can ask for a b2gbld upload endpoint, or swap to using ffxbld. There appear to be no remaining buildbot factory builds on ash, and in fact on m-c it's only hg-bundle, xulrunner, and l10n dep jobs which still reference 'stage.mozilla.org' explicitly. None of those are enabled on ash.

This is waiting on updated known_hosts being deployed on enough slaves to avoid false upload failures.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Pushed that to ash - https://hg.mozilla.org/projects/ash/rev/917678add467

Comment 3

[Chris AtLee [:catlee]]

Uploads look OK, but the new post_upload still isn't printing out the URLs for the uploaded resources.

Comment 4

[Jeremy Orem [:oremj]]

Can you give me the command you ran and the output that it generated?

Comment 5

[Chris AtLee [:catlee]]

The log is here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ash-linux-debug/1438229081/ash-linux-debug-bm71-build1-build0.txt.gz

The command we ran was:
post_upload.py --tinderbox-builds-dir ash-linux-debug -p firefox -i 20150729210441 --revision 917678add467 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds

Comment 6

[Jeremy Orem [:oremj]]

The --release-to-tinderbox-dated-builds command was broken. That should be working now. The --release-to-latest-tinderbox-builds, in the previous post_upload script, would symlink the dated builds directory to latest. Can we stop doing that? If not, how do we want to handle that operations since we can't atomically flip a directory.

Comment 7

[Chris AtLee [:catlee]]

We agreed earlier that instead of symlinking that you could copy the files into the latest directory.

Comment 8

[Jeremy Orem [:oremj]]

I can copy them over, but what about files that are already there? Should I leave them or delete everything in the directory?

Comment 9

[Chris AtLee [:catlee]]

Now getting these errors:

13:53:00     INFO -  /builds/slave/ash-lx-00000000000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/ash-lx-00000000000000000000000/build/src/build/upload.py --base-path ../../dist \
13:53:00     INFO -  	'../../dist/firefox-42.0a1.en-US.linux-i686.tar.bz2'   '../../dist/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi'  '../../dist/firefox-42.0a1.en-US.linux-i686.common.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.cppunittest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.xpcshell.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.mochitest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.reftest.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.web-platform.tests.zip' '../../dist/firefox-42.0a1.en-US.linux-i686.crashreporter-symbols.zip'  '../../dist//firefox-42.0a1.en-US.linux-i686.txt' '../../dist//firefox-42.0a1.en-US.linux-i686.json' '../../dist//firefox-42.0a1.en-US.linux-i686.mozinfo.json' '../../dist//test_packages.json' '../../dist/jsshell-linux-i686.zip'     ../../dist/host/bin/mar  ../../dist/host/bin/mbsdiff     \
13:53:00     INFO -  	'../../dist//firefox-42.0a1.en-US.linux-i686.checksums' '../../dist//firefox-42.0a1.en-US.linux-i686.checksums'.asc
13:56:58     INFO -  2015/07/30 20:56:58 putting /tmp/tmp.a43SLswCoz//firefox-42.0a1.en-US.linux-i686.tar.bz2 to net-mozaws-prod-delivery-firefox/pub/firefox/tinderbox-builds/ash-linux/1438229081/firefox-42.0a1.en-US.linux-i686.tar.bz2 err: NoSuchBucket: The specified bucket does not exist
13:56:58     INFO -  	status code: 404, request id: []
13:56:58     INFO -  2015/07/30 20:56:58 putting /tmp/tmp.a43SLswCoz/linux-i686/xpi/firefox-42.0a1.en-US.langpack.xpi to net-mozaws-prod-delivery-firefox/pub/firefox/tinderbox-builds/ash-linux/1438229081/firefox-42.0a1.en-US.langpack.xpi err: NoSuchBucket: The specified bucket does not exist
13:56:58     INFO -  	status code: 404, request id: []
13:56:59     INFO -  2015/07/30 20:56:59 putting /tmp/tmp.a43SLswCoz//firefox-42.0a1.en-US.linux-i686.common.tests.zip to net-mozaws-prod-delivery-firefox/pub/firefox/tinderbox-builds/ash-linux/1438229081/firefox-42.0a1.en-US.linux-i686.common.tests.zip err: NoSuchBucket: The specified bucket does not exist
13:56:59     INFO -  	status code: 404, request id: []

Comment 10

[Jeremy Orem [:oremj]]

You'll need the bucket-prefix flag we talked about earlier. --bucket-prefix "net-mozaws-stage-delivery"

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Merged m-c to ash, and added the bucket prefix, at https://hg.mozilla.org/projects/ash/rev/b3432877c7fe.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

We should turn on other jobs to make ash like m-c, eg nightlies, l10n, spidermonkey and other projects, in order to flush out ssh/scp/rsync usage. Transition to the new host is slated for mid-October.

Comment 13

[Rail Aliiev [:rail]]

Attachment: ash-buildbot-configs.diff

config diff can be viewed at https://gist.github.com/rail/2fc97bc7add5573af0ff

There are some unused variables, but cleaning them up would be another bug. If you prefer we can tweak some variables, like enable_nightly_everytime, enable_weekly_bundle (shouldn't we use bundleclone?).

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8659885 [details] [diff] [review]
ash-buildbot-configs.diff

Lets either back off on periodic_start_hours now or soon, unless those builds are checking for pushes as well as watching the clock. Strange that dep_signing_servers is being unset for debug builds.

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8659885 [details] [diff] [review]
ash-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/eb9fa2c84aee

pgo strategy set to per-checkin

Comment 16

[Rail Aliiev [:rail]]

pushed to production 
https://hg.mozilla.org/build/buildbot-configs/rev/ed59fecd49f1

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

I merged inbound to ash to see how we go - https://treeherder.mozilla.org/#/jobs?repo=ash&revision=4089002b3fae

Comment 18

[Rail Aliiev [:rail]]

I investigated multiple options to figure out what would be the optimal instance type for the upload host.


One of ideas was trying to simulate load comparable to what we have now.

First step was to figure out the current upload rates. I tried to use graphite, but the data is to coarse - tx/rx rates are not what we need.

I took this approach to figure out the upload rates we experience:

* find all files modified in particular period of time. I applied this to Firefox, Fennec and B2G files living on stage.m.o, modified within 24h for a busy day with multiple releases in fly (around Sep 15)
* Generate time series and analyze the rates. In our case the max is most important because we have to plan for peaks.

The results are the below:

30s max: 874 Mbps
1m max: 658 Mbps
3m max: 477 Mbps
5m max: 439 Mbps
10m max: 378 Mbps
1h max: 320 Mbps

Load simulation is a bit tricky task which may take a lot of resources. We thought that we could use taskcluster to spin up a lot of clients and upload some files. This will require some extra work to prep proper images with all needed secrets baked in and write custom scripts to generate traffic.

From our past experience with proxxy, we will need quite a beefy instance (assuming we can't use multiple instances in parallel) to meet the needed network performance. Per http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html m3.2xlarge might be what we need.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

rail, could you do what's necessary to turn on funsize for ash ?

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

I've pushed to ash with fixes for
* desktop balrog submission (replace archive.m.o with CDN instead of ftp.m.o), would have probably hit this on android too
* android multi-locale's missing config

See https://treeherder.mozilla.org/#/jobs?repo=ash&revision=9595d85e38b8.

B2G builds will still be busted uploading to the new system, probably because upload.ffxbld.stage needs to be allowed to write to /pub/b2g/.

Comment 21

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #19)
> rail, could you do what's necessary to turn on funsize for ash ?

You can add ash to https://github.com/mozilla/funsize/blob/master/funsize/worker.py#L22. Deployment is a bit tricky (need to document it..), I can help with that.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Thanks, https://github.com/mozilla/funsize/pull/22

Comment 23

[Rail Aliiev [:rail]]

Deployed new funsize:

remote:   https://hg.mozilla.org/build/puppet/rev/28303fd6d4b7
remote:   https://hg.mozilla.org/build/puppet/rev/61152257c0f7

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Current status: everything is green on ash except
* public b2g uploads, the upload host isn't allowed access to b2g/, bug 1211374
* b2g manifests will fail to upload after the switch, bug 1211371
* hazard builds do their own upload, bug 1211402
* the blocklist updates will break on esr38 when we lose tinderbox-builds/foo/latest, and have no nightlies, bug 1211770

(In reply to Rail Aliiev [:rail] from comment #23)
> Deployed new funsize:

Not sure if not working, or needs more nightlies to be able to make progress. In retrospect we only need to generate partials one build back.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Still using ash to finish up the source manifests and socorro json files, and then possibly for redoing the pvtbuilds uploads.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Finished using ash a long time ago. https://github.com/mozilla-releng/funsize/pull/41/files to celan up the funsize config.