1206041 - Teach release automation that the hosts for upload and download can be different

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We have stage.m.o for both ssh uploads and http(s) downloads, but after moving to S3 we'll have different hosts for each job.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] v1

There's several things going on here
* the new staging server for upload requires a --bucket-prefix to talk to the right S3 bucket, so we need to pass this around a lot of places (thought this was going to be less, otherwise I'd have asked oremj to fix that)
* those are defined in attachment 8651057 [details] [diff] [review], then used in several places
* ReleaseBuildFactory gets an ftpServer arg, so that it can download previous releases for partial mar generation (not for long, long live funsize!)
* make sure we don't request paths with // in them. Apache ignored that, the new system doesn't (eg oldCandidatesDir)
* stop uploading info.txt files in buildN dir for fennec, TODO - adjust the Socorro ftp scraper to cope
* stop using hardcoded post_upload commands in the source factory etc
* ReleaseUpdatesFactory no longer needs a stagingServer arg, it points at ftpServer for all downloads
* pass a new --ftp-server arg to l10n repacks, so they can get the en-US build and old complete mars

In the release config we'll have this for staging:
releaseConfig['ftpServer']           = 'ftp.stage.mozaws.net'
releaseConfig['stagingServer']       = 'upload.ffxbld.productdelivery.stage.mozaws.net'

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] v1

* Patcher land:
 * lots of renaming of stagingServer to ftpServer
 * more removals of // in request paths
 * use candidateDir rather than generate it again in many places
 * dump the ancient and crappy tests
* Balrog land:
 * renaming
 * we're going to use <app>/candidates/ and drop the symlinks in <app>/nightly that point back there, so drop the mobile hack in _getFileUrls()
* L10n land ('tis a mythical place full of dragons):
 * avoid // some more when getting mar and mbsdiff
 * add a DOWNLOAD_HOST variable for lightning, since that has yet another way of setting the EN_US url
* teach tools code about --bucket-prefix too

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8662896 [details] [diff] [review]
[buildbotcustom] v1

Review of attachment 8662896 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8662901 [details] [diff] [review]
[tools] v1

Review of attachment 8662901 [details] [diff] [review]:
-----------------------------------------------------------------

::: release/patcher-config-bump.pl
@@ +17,5 @@
>  # we disable this for bug 498273
>  # my $RELEASE_CANDIDATE_CHANNELS = ['betatest', 'DisableCompleteJump'];
>  
>  my %config;
> +my @DEFAULT_PLATFORMS = ('linux', 'linux64', 'macosx64', 'win32');

Feel free to add win64 as well

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8662901 [details] [diff] [review]
[tools] v1

https://hg.mozilla.org/build/tools/rev/b2d4cac21f15

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8662896 [details] [diff] [review]
[buildbotcustom] v1

https://hg.mozilla.org/build/buildbotcustom/rev/300082f07673
https://hg.mozilla.org/build/buildbotcustom/rev/1a685a2b7206

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

A few staging changes that got lost along the way, and busted the 42.0b5 updates builder with

mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/rel-m-beta-fx_beta_upds-000000/. --unpriv --shell '/usr/bin/env HG_SHARE_BA... perl tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 42.0b5 -a 42.0 -o 42.0b4 -b 1 -c tools/release/patcher-configs/mozBeta-branch-patcher2.cfg -f ftp.mozilla.org -d download.mozilla.org -l shipped-locales --partial-version 42.0b4 --partial-version 42.0b2 --partial-version 42.0b3 --platform linux --platform linux64 --platform macosx64 --platform win32 --platform win64'
 in dir /builds/slave/rel-m-beta-fx_beta_upds-000000/. (timeout 1200 secs)
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
Global symbol "$stagingServer" requires explicit package name at tools/release/patcher-config-bump.pl line 307.
Global symbol "$stagingServer" requires explicit package name at tools/release/patcher-config-bump.pl line 368.
Execution of tools/release/patcher-config-bump.pl aborted due to compilation errors.
State Changed: unlock buildroot


Landed this fix https://hg.mozilla.org/build/tools/rev/131d55ea931d

And this for the review comment about win64 I missed, https://hg.mozilla.org/build/tools/rev/1e67d3d9c321

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Fix urls in release block

We got this for 42.0b5 build1 in http://hg.mozilla.org/build/tools/rev/f83dced8be55:

+            <42.0b5>
+                checksumsurl   http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/42.0b5-candidates/build1build1/%platform%/%locale%/firefox-42.0b5.checksums
+                completemarurl   http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/42.0b5-candidates/build1build1/update/%platform%/%locale%/firefox-42.0b5.complete.mar

This fixes up the 'build1build1' for correctness's sake, even though I don't think we use these for anything.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8671767 [details] [diff] [review]
[tools] Fix urls in release block

https://hg.mozilla.org/build/tools/rev/323898b72cf8