1213721 - Tracking bug for migration from ftp.m.o to S3 (firefox, mobile, thunderbird)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This is the get-off-the-netapp move from ftp cluster in SCL3, moving onto AWS S3. Cloud Ops is driving this, we need to adjust where we upload things.

We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on Oct 13.

Bug 1186297 has been tracking the CI side, but I'll add other bugs here.
Bug 1165405 has all the bugs for release automation fixes.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Switch CI for firefox, mobile, thunderbird but not b2g

tl;dr stage.mozilla.org --> upload.ffxbld.productdelivery.prod.mozaws.net. This mainly affects branches like beta, release, and esr38 where we're not using mozharness.

We'll do b2g a little later, need to sort out things like bug 1211371 and bug 1211374 first. Fuzzing moves to pvtbuilds2 because it has private stuff, that gets sorted out later too. Drops mozilla.org in paths because that's being deprecated (but will still work via a 302), and swaps from ftp.m.o to archive.m.o (the new canonical domain).

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Unbuffered output for make upload

Between a local python doing uploads to the server, a python call to post_upload on the server, and probably buildbot stdin/stdout handling, we routinely end up with output mangled all together. This seems to be worse with the new system, but this patch solves the problem.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] mozharness config changes

The most important parts of this are configs/builds/build_pool_specifics.py and configs/single_locale/production.py, where the upload host is changing for Firefox desktop and mobile for all the branches except beta/release/esr38.

Other notable things
* the new upload host is going to respond with archive.m.o urls after post_upload, so translate that to the CDN for updates (mobile only given funsize for desktop)
* a bunch of noise setting up ash l10n for mobile
* s,(stage|ftp).mozilla.org/pub/mozilla.org,archive.mozilla.org/pub,g

This will need to land on a lot of branches when we transition over.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8672536 [details] [diff] [review]
[gecko] mozharness config changes

Review of attachment 8672536 [details] [diff] [review]:
-----------------------------------------------------------------

OOOOOH!

::: testing/mozharness/configs/single_locale/ash_android-api-11.py
@@ +2,4 @@
>  MOZ_UPDATE_CHANNEL = "nightly"
>  MOZILLA_DIR = BRANCH
>  OBJDIR = "obj-l10n"
> +EN_US_BINARY_URL = "http://archive.mozilla.org/pub/mobile/nightly/latest-%s-android-api-11/en-US" % (BRANCH)

A nit. Can you either remove the parenthesis around BRANCH or add a trailing coma to make it a tuple. Something like:
  ... % (BRANCH,)

Comment 5

[Mike Hommey [:glandium]]

Comment on attachment 8672532 [details] [diff] [review]
[gecko] Unbuffered output for make upload

Review of attachment 8672532 [details] [diff] [review]:
-----------------------------------------------------------------

Kind of gross, but meh. I guess at some point, we should force make to do the right thing, since make 4 now supports that.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8672532 [details] [diff] [review]
[gecko] Unbuffered output for make upload

https://hg.mozilla.org/integration/mozilla-inbound/rev/689d1b250ba0

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] mozharness config changes, v2

With '% (BRANCH)'  --> '% BRANCH' changes. Carrying rail's review.

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/689d1b250ba0

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/689d1b250ba0

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c7cedbb6e3fb

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/aa65b3b28dcf

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/169f5981e5c0

Comment 13

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/605de27d4e7f

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/aa65b3b28dcf

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c7cedbb6e3fb

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8672505 [details] [diff] [review]
[buildbot-configs] Switch CI for firefox, mobile, thunderbird but not b2g

Landed yesterday http://hg.mozilla.org/build/buildbot-configs/rev/f500b8d4b883

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8675927 [details] [diff] [review]
[gecko] mozharness config changes, v2

Landed yesterday on inbound at c7cedbb6e3fb, as well fx-team, b2g-inbound and aurora.

TODO - try to merge to beta/release/esr38, although we don't use mozharness for ci builds there. Maybe people push to try occasionally, although I'm not sure that works.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8675927 [details] [diff] [review]
[gecko] mozharness config changes, v2

Merged to release branches:

https://hg.mozilla.org/releases/mozilla-beta/rev/920d1c0c8d1d
https://hg.mozilla.org/releases/mozilla-release/rev/5bb35852d220
http://hg.mozilla.org/releases/mozilla-esr38/rev/faadd70c8d24 (default)
http://hg.mozilla.org/releases/mozilla-esr38/rev/9a453c7474a2 (tb branch)

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

9a453.. should be http://hg.mozilla.org/releases/mozilla-esr38/rev/9513063f2630

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Should land on the project branches too, at least date is still uploading to scl3.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/projects/date/rev/eb1d231160bf
https://hg.mozilla.org/projects/pine/rev/04724fcfebcc

Everything else is updated (ash, elm etc), or unowned/inactive.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Thunderbird fixup

I forgot to cover the Thunderbird production case, so that's trying to use to upload.ffxbld instead of upload.tbirdbird. Also update download_base_url to use the new domain and drop mozilla.org from the path. I updated the try values, but I'm pretty sure it's not used there.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8679738 [details] [diff] [review]
[buildbot-configs] Thunderbird fixup

https://hg.mozilla.org/build/buildbot-configs/rev/229776123ec4
https://hg.mozilla.org/build/buildbot-configs/rev/db1761df121c

Reconfig at 1700 Pacific picked this up.

Comment 24

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.