Serve nightly & aurora updates from the CDN

RESOLVED FIXED

Status

Release Engineering
General Automation
--
critical
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: nthomas, Assigned: rail)

Tracking

unspecified
x86
All
Dependency tree / graph

Firefox Tracking Flags

(firefox37 fixed, firefox38 fixed, firefox39 fixed)

Details

Attachments

(8 attachments, 6 obsolete attachments)

1.87 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
9.86 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.03 KB, patch
rail
: review+
Details | Diff | Splinter Review
1.94 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.27 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
5.63 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
309 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.85 KB, patch
rail
: review+
mshal
: checked-in+
Details | Diff | Splinter Review
(Reporter)

Description

3 years ago
We serve more than 1.5 TB/day for .mar files on ftp.m.o, which averages to 200 Mbps. The lions share of that is likely to be updates for nightly and aurora. Since we are currently having trouble with Zeus traffic levels I'm proposing we move that to the CDN, and bypassing bouncer. 

This means we just point to download.cdn.mozilla.net instead of ftp.mozilla.org when submitting into Balrog, and acknowledge that we won't have any logs as a result. I don't think setting up bouncer products for each nightly just to get hits on download.m.o is useful, given the data is not processed in any accessible way.

We'll have to add download.cdn.mozilla.net to the domain_whitelist in balrog.ini and admin.ini.
Getting this done for the current updates has become high priority, doing that right now. Hopefully it will be trivial to enable it for future updates afterwards.
Assignee: nobody → bhearsum
Severity: normal → critical
Depends on: 1145186
Okay, Nightly and Aurora traffic should be on the CDN. Here's what we did:
* Downloaded the Firefox-mozilla-{central,aurora}-nightly-latest release blobs from Balrog.
* Copied them to -cdn versions.
* s/ftp.mozilla.org/download.cdn.mozilla.net/g in them
* Uploaded to Balrog with the new names
* Repointed nightly and aurora rules at them

With this in place, all nightly and aurora Firefox updates should be on the CDN instead of FTP. Because we've used a new release name, the next set of builds will NOT be automatically published. We need to do one of the following to get the next set published:
1) Adjust our CI to publish nightlies that point at download.cdn.mozilla.net automatically (instead of ftp.mozilla.org). Repoint nightly/aurora rules at original release blobs after the next set completes (making sure to wait for at least some l10n jobs).
2) Repeat the steps above after tomorrow's nightlies finish.
(Assignee)

Comment 3

3 years ago
Automation uses URLs printed by post_upload.py. We can:

1) change http://hg.mozilla.org/build/tools/file/b5405137c3dc/stage/production.ini#l15 and point ALL bits to CDN. This means that the tests will be hitting CDN too

2) add hacks to post_upload.py to treat MAR files differently (have different domain for them)

3) add hacks to automation to replace ftp.mozilla.org with cdn domain

The first option is the cleanest one, not sure how comfortable we with serving ALL nightly files form CDNs.
(Assignee)

Comment 4

3 years ago
Created attachment 8580135 [details] [diff] [review]
cdn.diff

If we are comfortable with option 1, we can use this patch.

The file needs to be copied to the productdelivery svn module by someone with access (I lost mine at some point).
Attachment #8580135 - Flags: review?(bhearsum)
Comment on attachment 8580135 [details] [diff] [review]
cdn.diff

Review of attachment 8580135 [details] [diff] [review]:
-----------------------------------------------------------------

Do we know if our build and test machines can route to the CDNs? If so, how do we make sure it doesn't accidentally change in the future?

I'm still not sure it's great to move this traffic to the CDNs, especially on a whim. I'd be more comfortable if we could _only_ change what's in the blob....
(Assignee)

Comment 6

3 years ago
Comment on attachment 8580135 [details] [diff] [review]
cdn.diff

Just chatted with catlee. We came to the conclusion that option 3 would be clearer here.
Attachment #8580135 - Flags: review?(bhearsum)
(Assignee)

Comment 7

3 years ago
Created attachment 8580181 [details] [diff] [review]
replace_canonical_urls-tools.diff

Something like this maybe?
Attachment #8580181 - Flags: review?(bhearsum)
Comment on attachment 8580181 [details] [diff] [review]
replace_canonical_urls-tools.diff

Review of attachment 8580181 [details] [diff] [review]:
-----------------------------------------------------------------

Hiding this all the way down here kindof sucks. It would be nicer to do it at the script level if possible...

We should make sure that all of the files that we point at are on the CDNs. It looks like that's the case (eg, download.cdn.mozilla.net/pub/mozilla.org/b2g/nightly/2015-03-19-00-12-27-mozilla-b2g34_v2_1-flame-kk/b2g-flame-gecko-update.mar works), but there might be an edge case that I'm missing...
(Reporter)

Comment 9

3 years ago
Thanks for picking this work up. FWIW, I started on a patch pretty similar to this. Memory asserts that the origin server has access to all files, but there's a slight chance we might break things with future changes. No doubt it will change with s!ftp!S3! anyway.
(Assignee)

Comment 10

3 years ago
I updated the blobs like in comment#2 today.
(Assignee)

Comment 11

3 years ago
Created attachment 8580721 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

This is another, more indirect approach.
Attachment #8580181 - Attachment is obsolete: true
Attachment #8580181 - Flags: review?(bhearsum)
Attachment #8580721 - Flags: feedback?(bhearsum)
(Assignee)

Comment 12

3 years ago
Created attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff
Attachment #8580722 - Flags: feedback?(bhearsum)
(Assignee)

Updated

3 years ago
Blocks: 1146185
Tracking for 38 and 39.
status-firefox37: --- → affected
status-firefox38: --- → affected
tracking-firefox38: --- → +
tracking-firefox39: --- → +
(Reporter)

Comment 14

2 years ago
I've pointed Nightly and Aurora updates back to the latest builds, via ftp.m.o because we have more traffic headroom there now. We have a month to decide how to switch to the CDN.
(Assignee)

Comment 15

2 years ago
Created attachment 8581574 [details] [diff] [review]
replace_canonical_urls-tools.diff
Attachment #8580721 - Attachment is obsolete: true
Attachment #8580721 - Flags: feedback?(bhearsum)
Attachment #8581574 - Flags: feedback?(bhearsum)
(Assignee)

Comment 16

2 years ago
Created attachment 8581580 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

Moar test. Try in production? :)
Attachment #8581574 - Attachment is obsolete: true
Attachment #8581574 - Flags: feedback?(bhearsum)
Attachment #8581580 - Flags: review?(bhearsum)
(Assignee)

Updated

2 years ago
Attachment #8580722 - Flags: feedback?(bhearsum) → review?(bhearsum)
Comment on attachment 8581580 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

Review of attachment 8581580 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/balrog/submitter/cli.py
@@ +306,5 @@
>  class MultipleUpdatesNightlyMixin(object):
> +
> +    def _maybe_replace_canocical_url(self, url):
> +        # self._replace_canocical_url() and self.url_replacements come from
> +        # NightlySubmitterBase. Make sure they exist in concrete classes

This is pretty nasty....are you sure it's necessary? I don't see url_replacements ever getting passed to Release* classes.
Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff

Review of attachment 8580722 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/balrog/production.py
@@ +4,5 @@
>              'balrog_api_root': 'https://aus4-admin.mozilla.org/api',
>              'ignore_failures': False,
> +            'url_replacements': [
> +                ('http://ftp.mozilla.org/pub/mozilla.org',
> +                 'http://download.cdn.mozilla.net/pub/mozilla.org'),

Minor nit: this would read better with both URLs on the same line IMO. If we ever add a second one it will be harder to see the before and after URLs at a glance.
Attachment #8580722 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 19

2 years ago
Created attachment 8581592 [details] [diff] [review]
replace_canonical_urls-tools-2.diff

Fix a typo.

I ran this against localhost (nothing running) with the following patch applied https://gist.github.com/rail/5330ebb3d49da9deefb7 (just to  dump data):


python scripts/updates/balrog-submitter.py --build-properties ~/tmp/b.json --api-root https://localhost/api --username ffxbld -t nightly --credentials-file ~/tmp/oauth.txt --verbose --url-replacement  ftp.mozilla.org,download.cdn.mozilla.net                                                                                                                                                            
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar
{'product': u'Firefox', 'hashFunction': u'sha512', 'schema_version': 4, 'alias': 'null', 'copyTo': '["Firefox-mozilla-central-nightly-latest"]', 'version': u'39.0a1', 'data': '{"platformVersion": "39.0a1", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar", "hashValue": "a8ec9c0ba52cb153511c0842f8a2109d056978e93e0aeecb852ae2b93fbe3ff905de948814c34b195998f20eb7c013e7425a688eee1b4213e208b8aeb91be9ff", "from": "Firefox-mozilla-central-nightly-20150322030216", "filesize": 8244610}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar", "hashValue": "c4ea1e8548582691c49b36c9bf0d0ba5b5b22e9b2313486cac9926e9ab1d95cb297f32f755127f5043d92c0c9dcc7aa13166bafcd761b62a27fed97b64cd6b20", "from": "*", "filesize": 54133063}], "buildID": "20150323030203", "displayVersion": "39.0a1", "appVersion": "39.0a1"}'}
Attachment #8581580 - Attachment is obsolete: true
Attachment #8581580 - Flags: review?(bhearsum)
Attachment #8581592 - Flags: review?(bhearsum)
(Assignee)

Comment 20

2 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Comment on attachment 8581580 [details] [diff] [review]
> replace_canonical_urls-tools-1.diff
> 
> Review of attachment 8581580 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: lib/python/balrog/submitter/cli.py
> @@ +306,5 @@
> >  class MultipleUpdatesNightlyMixin(object):
> > +
> > +    def _maybe_replace_canocical_url(self, url):
> > +        # self._replace_canocical_url() and self.url_replacements come from
> > +        # NightlySubmitterBase. Make sure they exist in concrete classes
> 
> This is pretty nasty....are you sure it's necessary? I don't see
> url_replacements ever getting passed to Release* classes.

You mean I should remove the guards?
(Assignee)

Updated

2 years ago
Attachment #8580135 - Attachment is obsolete: true
(In reply to Rail Aliiev [:rail] from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> > Comment on attachment 8581580 [details] [diff] [review]
> > replace_canonical_urls-tools-1.diff
> > 
> > Review of attachment 8581580 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: lib/python/balrog/submitter/cli.py
> > @@ +306,5 @@
> > >  class MultipleUpdatesNightlyMixin(object):
> > > +
> > > +    def _maybe_replace_canocical_url(self, url):
> > > +        # self._replace_canocical_url() and self.url_replacements come from
> > > +        # NightlySubmitterBase. Make sure they exist in concrete classes
> > 
> > This is pretty nasty....are you sure it's necessary? I don't see
> > url_replacements ever getting passed to Release* classes.
> 
> You mean I should remove the guards?

I don't see any reason for them to exist, so I would hesistantly say yes. They don't actually affect any behaviour right now, right?
(Assignee)

Comment 22

2 years ago
Created attachment 8581713 [details] [diff] [review]
replace_canonical_urls-tools.diff

Interdiff https://gist.github.com/rail/b17f8011ba632ca6d4f8
Attachment #8581592 - Attachment is obsolete: true
Attachment #8581592 - Flags: review?(bhearsum)
Attachment #8581713 - Flags: review?(bhearsum)
Attachment #8581713 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 23

2 years ago
Comment on attachment 8581713 [details] [diff] [review]
replace_canonical_urls-tools.diff

https://hg.mozilla.org/build/tools/rev/789a3138bd4d
Attachment #8581713 - Flags: checked-in+
(Assignee)

Comment 24

2 years ago
Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff

https://hg.mozilla.org/build/mozharness/rev/a814625fcd39
Attachment #8580722 - Flags: checked-in+
Assignee: bhearsum → rail
(Assignee)

Comment 25

2 years ago
Created attachment 8581765 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

Make python 2.6 happier.
Attachment #8581765 - Flags: review?(bhearsum)
(Assignee)

Comment 26

2 years ago
Comment on attachment 8581765 [details] [diff] [review]
replace_canonical_urls-tools-1.diff

r=bhearsum in IRC
https://hg.mozilla.org/build/tools/rev/0d0c1165741c
Attachment #8581765 - Flags: review?(bhearsum)
Attachment #8581765 - Flags: review+
Attachment #8581765 - Flags: checked-in+
(Assignee)

Comment 27

2 years ago
Created attachment 8581836 [details] [diff] [review]
replace_canonical_urls-tools-2.diff

Of course, skipIf if 2.7+ only...
green: https://travis-ci.org/rail/build-tools/builds/55529377
Attachment #8581836 - Flags: review?(bhearsum)
Attachment #8581836 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 28

2 years ago
Comment on attachment 8581836 [details] [diff] [review]
replace_canonical_urls-tools-2.diff

https://hg.mozilla.org/build/tools/rev/bccc7e199158
Attachment #8581836 - Flags: checked-in+
mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production
(Assignee)

Comment 30

2 years ago
Created attachment 8582044 [details] [diff] [review]
configs-nighlty.diff

For nighlty and l10n which are not in mozharness yet
Attachment #8582044 - Flags: review?(bhearsum)
(Assignee)

Comment 31

2 years ago
Created attachment 8582046 [details] [diff] [review]
custom-nightly.diff
Attachment #8582046 - Flags: review?(bhearsum)
(Assignee)

Comment 32

2 years ago
I updated the blobs like in comment#2 today.
(Assignee)

Comment 33

2 years ago
Created attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff

In-tree bump
Attachment #8582388 - Flags: review?(bhearsum)
Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff

Review of attachment 8582388 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think you need review for mozharness version bumps, but r+ anyways.
Attachment #8582388 - Flags: review?(bhearsum) → review+
Attachment #8582046 - Flags: review?(bhearsum) → review+
Attachment #8582044 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 35

2 years ago
Comment on attachment 8582044 [details] [diff] [review]
configs-nighlty.diff

https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118
Attachment #8582044 - Flags: checked-in+
(Assignee)

Comment 36

2 years ago
Comment on attachment 8582046 [details] [diff] [review]
custom-nightly.diff

https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a
Attachment #8582046 - Flags: checked-in+
(Assignee)

Comment 37

2 years ago
Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff

mshal jinxed me in https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f :)
Attachment #8582388 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118
In production: https://hg.mozilla.org/build/buildbot-configs/rev/22f9faca403c
In production: https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a
(Assignee)

Comment 41

2 years ago
L10N repacks worked fine (from yesterday), Android worked fine, waiting for m-c en-US build.
(Assignee)

Comment 42

2 years ago
Just checked mac en-US nightly, the partial in Firefox-mozilla-central-nightly-20150325030206 looks good. The complete is pointing to https://queue.taskcluster.net/v1/task/dUi7oY11RxaXt7hjH_wE8A/artifacts/public/build/firefox-39.0a1.en-US.mac.complete.mar. Is this expected?
(Assignee)

Comment 43

2 years ago
http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-central%20nightly/builds/4 properties look interesting. None of the properties contain "https://queue.taskcluster.net/", but if you look to the script output, some URLs use ftp.m.o, some queue.t.n. balrog-submitter.py itself is a bit confused which one to use. :)

CCing Mike, he may have some info.
It looks like I might just need to add a 'write_to_file=True' argument to set_buildbot_property() when mozharness is uploading the arguments to TC. I'm going to test that out on maple...
Created attachment 8583847 [details] [diff] [review]
disable-mar-TC

This is untested at the moment, but it should avoid overriding completeMarUrl and partialMarUrl. We still need to keep the property conditions in there though to avoid stomping on packageUrl, so these are just renamed to completeMarUrlTC and partialMarUrlTC, which shouldn't be used by anything else.
Attachment #8583847 - Flags: review?(rail)
(Assignee)

Updated

2 years ago
Attachment #8583847 - Flags: review?(rail) → review+
Comment on attachment 8583847 [details] [diff] [review]
disable-mar-TC

https://hg.mozilla.org/build/mozharness/rev/db4fdf68f146
Attachment #8583847 - Flags: checked-in+
(Assignee)

Comment 47

2 years ago
Live!
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
I guess it is fixed for 38 & 37.
status-firefox37: affected → fixed
status-firefox38: affected → fixed
(In reply to Sylvestre Ledru [:sylvestre] from comment #48)
> I guess it is fixed for 38 & 37.

These flags really don't apply here...this isn't in-tree code that's being tweaked, and aurora and nightly are already at 39 & 40...
Looks like this was fixed while 39 was in nightly so we can call it fixed for 39+. Untracking.
status-firefox39: --- → fixed
tracking-firefox38: + → ---
tracking-firefox39: + → ---
You need to log in before you can comment on or make changes to this bug.