1144985 - Serve nightly & aurora updates from the CDN

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We serve more than 1.5 TB/day for .mar files on ftp.m.o, which averages to 200 Mbps. The lions share of that is likely to be updates for nightly and aurora. Since we are currently having trouble with Zeus traffic levels I'm proposing we move that to the CDN, and bypassing bouncer. 

This means we just point to download.cdn.mozilla.net instead of ftp.mozilla.org when submitting into Balrog, and acknowledge that we won't have any logs as a result. I don't think setting up bouncer products for each nightly just to get hits on download.m.o is useful, given the data is not processed in any accessible way.

We'll have to add download.cdn.mozilla.net to the domain_whitelist in balrog.ini and admin.ini.

Comment 1

[[e-mail] (:bhearsum)]

Getting this done for the current updates has become high priority, doing that right now. Hopefully it will be trivial to enable it for future updates afterwards.

Comment 2

[[e-mail] (:bhearsum)]

Okay, Nightly and Aurora traffic should be on the CDN. Here's what we did:
* Downloaded the Firefox-mozilla-{central,aurora}-nightly-latest release blobs from Balrog.
* Copied them to -cdn versions.
* s/ftp.mozilla.org/download.cdn.mozilla.net/g in them
* Uploaded to Balrog with the new names
* Repointed nightly and aurora rules at them

With this in place, all nightly and aurora Firefox updates should be on the CDN instead of FTP. Because we've used a new release name, the next set of builds will NOT be automatically published. We need to do one of the following to get the next set published:
1) Adjust our CI to publish nightlies that point at download.cdn.mozilla.net automatically (instead of ftp.mozilla.org). Repoint nightly/aurora rules at original release blobs after the next set completes (making sure to wait for at least some l10n jobs).
2) Repeat the steps above after tomorrow's nightlies finish.

Comment 3

[Rail Aliiev [:rail]]

Automation uses URLs printed by post_upload.py. We can:

1) change http://hg.mozilla.org/build/tools/file/b5405137c3dc/stage/production.ini#l15 and point ALL bits to CDN. This means that the tests will be hitting CDN too

2) add hacks to post_upload.py to treat MAR files differently (have different domain for them)

3) add hacks to automation to replace ftp.mozilla.org with cdn domain

The first option is the cleanest one, not sure how comfortable we with serving ALL nightly files form CDNs.

Comment 4

[Rail Aliiev [:rail]]

Attachment: cdn.diff

If we are comfortable with option 1, we can use this patch.

The file needs to be copied to the productdelivery svn module by someone with access (I lost mine at some point).

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 8580135 [details] [diff] [review]
cdn.diff

Review of attachment 8580135 [details] [diff] [review]:
-----------------------------------------------------------------

Do we know if our build and test machines can route to the CDNs? If so, how do we make sure it doesn't accidentally change in the future?

I'm still not sure it's great to move this traffic to the CDNs, especially on a whim. I'd be more comfortable if we could _only_ change what's in the blob....

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8580135 [details] [diff] [review]
cdn.diff

Just chatted with catlee. We came to the conclusion that option 3 would be clearer here.

Comment 7

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools.diff

Something like this maybe?

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8580181 [details] [diff] [review]
replace_canonical_urls-tools.diff

Review of attachment 8580181 [details] [diff] [review]:
-----------------------------------------------------------------

Hiding this all the way down here kindof sucks. It would be nicer to do it at the script level if possible...

We should make sure that all of the files that we point at are on the CDNs. It looks like that's the case (eg, download.cdn.mozilla.net/pub/mozilla.org/b2g/nightly/2015-03-19-00-12-27-mozilla-b2g34_v2_1-flame-kk/b2g-flame-gecko-update.mar works), but there might be an edge case that I'm missing...

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for picking this work up. FWIW, I started on a patch pretty similar to this. Memory asserts that the origin server has access to all files, but there's a slight chance we might break things with future changes. No doubt it will change with s!ftp!S3! anyway.

Comment 10

[Rail Aliiev [:rail]]

I updated the blobs like in comment#2 today.

Comment 11

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools-1.diff

This is another, more indirect approach.

Comment 12

[Rail Aliiev [:rail]]

Attachment: balrog_replace_urls-mozharness.diff

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 38 and 39.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I've pointed Nightly and Aurora updates back to the latest builds, via ftp.m.o because we have more traffic headroom there now. We have a month to decide how to switch to the CDN.

Comment 15

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools.diff

Comment 16

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools-1.diff

Moar test. Try in production? :)

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 8581580 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

Review of attachment 8581580 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/balrog/submitter/cli.py
@@ +306,5 @@
>  class MultipleUpdatesNightlyMixin(object):
> +
> +    def _maybe_replace_canocical_url(self, url):
> +        # self._replace_canocical_url() and self.url_replacements come from
> +        # NightlySubmitterBase. Make sure they exist in concrete classes

This is pretty nasty....are you sure it's necessary? I don't see url_replacements ever getting passed to Release* classes.

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff

Review of attachment 8580722 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/balrog/production.py
@@ +4,5 @@
>              'balrog_api_root': 'https://aus4-admin.mozilla.org/api',
>              'ignore_failures': False,
> +            'url_replacements': [
> +                ('http://ftp.mozilla.org/pub/mozilla.org',
> +                 'http://download.cdn.mozilla.net/pub/mozilla.org'),

Minor nit: this would read better with both URLs on the same line IMO. If we ever add a second one it will be harder to see the before and after URLs at a glance.

Comment 19

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools-2.diff

Fix a typo.

I ran this against localhost (nothing running) with the following patch applied https://gist.github.com/rail/5330ebb3d49da9deefb7 (just to  dump data):


python scripts/updates/balrog-submitter.py --build-properties ~/tmp/b.json --api-root https://localhost/api --username ffxbld -t nightly --credentials-file ~/tmp/oauth.txt --verbose --url-replacement  ftp.mozilla.org,download.cdn.mozilla.net                                                                                                                                                            
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar
{'product': u'Firefox', 'hashFunction': u'sha512', 'schema_version': 4, 'alias': 'null', 'copyTo': '["Firefox-mozilla-central-nightly-latest"]', 'version': u'39.0a1', 'data': '{"platformVersion": "39.0a1", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar", "hashValue": "a8ec9c0ba52cb153511c0842f8a2109d056978e93e0aeecb852ae2b93fbe3ff905de948814c34b195998f20eb7c013e7425a688eee1b4213e208b8aeb91be9ff", "from": "Firefox-mozilla-central-nightly-20150322030216", "filesize": 8244610}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar", "hashValue": "c4ea1e8548582691c49b36c9bf0d0ba5b5b22e9b2313486cac9926e9ab1d95cb297f32f755127f5043d92c0c9dcc7aa13166bafcd761b62a27fed97b64cd6b20", "from": "*", "filesize": 54133063}], "buildID": "20150323030203", "displayVersion": "39.0a1", "appVersion": "39.0a1"}'}

Comment 20

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Comment on attachment 8581580 [details] [diff] [review]
> replace_canonical_urls-tools-1.diff
> 
> Review of attachment 8581580 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: lib/python/balrog/submitter/cli.py
> @@ +306,5 @@
> >  class MultipleUpdatesNightlyMixin(object):
> > +
> > +    def _maybe_replace_canocical_url(self, url):
> > +        # self._replace_canocical_url() and self.url_replacements come from
> > +        # NightlySubmitterBase. Make sure they exist in concrete classes
> 
> This is pretty nasty....are you sure it's necessary? I don't see
> url_replacements ever getting passed to Release* classes.

You mean I should remove the guards?

Comment 21

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> > Comment on attachment 8581580 [details] [diff] [review]
> > replace_canonical_urls-tools-1.diff
> > 
> > Review of attachment 8581580 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: lib/python/balrog/submitter/cli.py
> > @@ +306,5 @@
> > >  class MultipleUpdatesNightlyMixin(object):
> > > +
> > > +    def _maybe_replace_canocical_url(self, url):
> > > +        # self._replace_canocical_url() and self.url_replacements come from
> > > +        # NightlySubmitterBase. Make sure they exist in concrete classes
> > 
> > This is pretty nasty....are you sure it's necessary? I don't see
> > url_replacements ever getting passed to Release* classes.
> 
> You mean I should remove the guards?

I don't see any reason for them to exist, so I would hesistantly say yes. They don't actually affect any behaviour right now, right?

Comment 22

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools.diff

Interdiff https://gist.github.com/rail/b17f8011ba632ca6d4f8

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8581713 [details] [diff] [review]
replace_canonical_urls-tools.diff

https://hg.mozilla.org/build/tools/rev/789a3138bd4d

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff

https://hg.mozilla.org/build/mozharness/rev/a814625fcd39

Comment 25

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools-1.diff

Make python 2.6 happier.

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8581765 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

r=bhearsum in IRC
https://hg.mozilla.org/build/tools/rev/0d0c1165741c

Comment 27

[Rail Aliiev [:rail]]

Attachment: replace_canonical_urls-tools-2.diff

Of course, skipIf if 2.7+ only...
green: https://travis-ci.org/rail/build-tools/builds/55529377

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8581836 [details] [diff] [review]
replace_canonical_urls-tools-2.diff

https://hg.mozilla.org/build/tools/rev/bccc7e199158

Comment 29

[Jordan Lund (:jlund)]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment 30

[Rail Aliiev [:rail]]

Attachment: configs-nighlty.diff

For nighlty and l10n which are not in mozharness yet

Comment 31

[Rail Aliiev [:rail]]

Attachment: custom-nightly.diff

Comment 32

[Rail Aliiev [:rail]]

I updated the blobs like in comment#2 today.

Comment 33

[Rail Aliiev [:rail]]

Attachment: mozilla-inbound_bump_mozharness_rev.diff

In-tree bump

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff

Review of attachment 8582388 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think you need review for mozharness version bumps, but r+ anyways.

Comment 35

[Rail Aliiev [:rail]]

Comment on attachment 8582044 [details] [diff] [review]
configs-nighlty.diff

https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 8582046 [details] [diff] [review]
custom-nightly.diff

https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a

Comment 37

[Rail Aliiev [:rail]]

Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff

mshal jinxed me in https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f :)

Comment 38

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118

Comment 39

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/22f9faca403c

Comment 40

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a

Comment 41

[Rail Aliiev [:rail]]

L10N repacks worked fine (from yesterday), Android worked fine, waiting for m-c en-US build.

Comment 42

[Rail Aliiev [:rail]]

Just checked mac en-US nightly, the partial in Firefox-mozilla-central-nightly-20150325030206 looks good. The complete is pointing to https://queue.taskcluster.net/v1/task/dUi7oY11RxaXt7hjH_wE8A/artifacts/public/build/firefox-39.0a1.en-US.mac.complete.mar. Is this expected?

Comment 43

[Rail Aliiev [:rail]]

http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-central%20nightly/builds/4 properties look interesting. None of the properties contain "https://queue.taskcluster.net/", but if you look to the script output, some URLs use ftp.m.o, some queue.t.n. balrog-submitter.py itself is a bit confused which one to use. :)

CCing Mike, he may have some info.

Comment 44

[Michael Shal [:mshal]]

It looks like I might just need to add a 'write_to_file=True' argument to set_buildbot_property() when mozharness is uploading the arguments to TC. I'm going to test that out on maple...

Comment 45

[Michael Shal [:mshal]]

Attachment: disable-mar-TC

This is untested at the moment, but it should avoid overriding completeMarUrl and partialMarUrl. We still need to keep the property conditions in there though to avoid stomping on packageUrl, so these are just renamed to completeMarUrlTC and partialMarUrlTC, which shouldn't be used by anything else.

Comment 46

[Michael Shal [:mshal]]

Comment on attachment 8583847 [details] [diff] [review]
disable-mar-TC

https://hg.mozilla.org/build/mozharness/rev/db4fdf68f146

Comment 47

[Rail Aliiev [:rail]]

Live!

Comment 48

[Sylvestre Ledru [:Sylvestre]]

I guess it is fixed for 38 & 37.

Comment 49

[[e-mail] (:bhearsum)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #48)
> I guess it is fixed for 38 & 37.

These flags really don't apply here...this isn't in-tree code that's being tweaked, and aurora and nightly are already at 39 & 40...

Comment 50

[Liz Henry (:lizzard) (relman/hg->git project)]

Looks like this was fixed while 39 was in nightly so we can call it fixed for 39+. Untracking.