1223633 - Tracking bug for migration from ftp.m.o to S3 for SeaMonkey

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Edmund Wong (:ewong)]

We're migrating off ftp.mo to S3 (pending approval, just getting this bug
up) on the 17th November, so there's a bunch of stuff that needs to be done. 

+++ This bug was initially created as a clone of Bug #1213721 +++

This is the get-off-the-netapp move from ftp cluster in SCL3, moving onto AWS S3. Cloud Ops is driving this, we need to adjust where we upload things.

We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on Oct 13.

Bug 1186297 has been tracking the CI side, but I'll add other bugs here.
Bug 1165405 has all the bugs for release automation fixes.

Comment 1

[Tony Mechelynck [:tonymec]]

(In reply to Edmund Wong (:ewong) from comment #0)
> We're migrating off ftp.mo [...]

Does this mean that the download URL for trunk and aurora nightlies will change, or will they be redirected or redefined to wherever the nightlies will henceforward be published?

> We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on
> Oct 13.
[...]

I suppose you mean November 13 (today), not 11 months from now?

Comment 2

[Edmund Wong (:ewong)]

(In reply to Tony Mechelynck [:tonymec] from comment #1)
> (In reply to Edmund Wong (:ewong) from comment #0)
> > We're migrating off ftp.mo [...]
> 
> Does this mean that the download URL for trunk and aurora nightlies will
> change, or will they be redirected or redefined to wherever the nightlies
> will henceforward be published?
> 

Downloading will most likely be from archive.mozilla.org (though I'm not
100% sure on this).  This mostly will affect how the releng side uploads
the releases/tinderbox stuff because upload and downloading 'stuff' will
be from different hosts. 

> > We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on
> > Oct 13.
> [...]
> 
> I suppose you mean November 13 (today), not 11 months from now?

No,  that was the original scheduled date I believe.  It's now due to be
moved on Nov 17, which is three days from now.

Comment 3

[Edmund Wong (:ewong)]

Attachment: [configs] proposed wip patch (v1)

Comment 4

[Edmund Wong (:ewong)]

Due to the differences between SeaMonkey's releng and Firefox's releng, 
copying Firefox's changes to our buildbotcustom code isn't the right
approach.

Wouldn't it be easier if we just find out the needed urls (which I think
my buildbot-config changes cover), the needed credentials and the
needed procedures to enable our releng process to upload to the staging?
Right now, Firefox uses mozharness to do the general release stuff,
specifically the push-candidate-to-releases.py.  

Callek, any ideas how I should proceed?

Comment 5

[Edmund Wong (:ewong)]

Attachment: [configs] proposed patch (v2)

Comment 6

[Edmund Wong (:ewong)]

Attachment: [tools] change paths to reflect the ftp.mo->S3 changes. [checked-in]

[going for a post-land-review] as it will help with the manually patched
configs repo using the configs patch.

Comment 7

[Edmund Wong (:ewong)]

Comment on attachment 8690531 [details] [diff] [review]
[tools] change paths to reflect the ftp.mo->S3 changes. [checked-in]

Pushed to tools repo:
 https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/164592942e31

Comment 8

[Edmund Wong (:ewong)]

Attachment: [puppet] add upload host to knownhosts template. (v1)[checked-in]

for a post-land-review

Comment 9

[Edmund Wong (:ewong)]

Comment on attachment 8690647 [details] [diff] [review]
[puppet] add upload host to knownhosts template. (v1)[checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/e1d0fff5265a

Comment 10

[Edmund Wong (:ewong)]

Attachment: [tools] Fixed upload paths in post_upload.py [checked-in]

Comment 11

[Edmund Wong (:ewong)]

Comment on attachment 8691666 [details] [diff] [review]
[tools] Fixed upload paths in post_upload.py [checked-in]

Pushed to tools repo:
https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/752df09c4a82

for post-land-review

Comment 12

[Edmund Wong (:ewong)]

Attachment: [puppet] setup s3 credentials (v1)

Comment 13

[Edmund Wong (:ewong)]

Attachment: [puppet] support s3 credentials.

Comment 14

[Edmund Wong (:ewong)]

Attachment: [puppet] support s3 credentials. (v3) [checked-in]

Comment 15

[Edmund Wong (:ewong)]

Comment on attachment 8728274 [details] [diff] [review]
[puppet] support s3 credentials. (v3) [checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/416b6a95f4d6

Comment 16

[Edmund Wong (:ewong)]

Attachment: [puppet] support s3 credentials fix (v1) [checked-in]

Comment 17

[Edmund Wong (:ewong)]

Comment on attachment 8728278 [details] [diff] [review]
[puppet] support s3 credentials fix (v1) [checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/2f33a204fe67

Comment 18

[Edmund Wong (:ewong)]

the scripts which were located on stage.mo have been 'translated' to using
the AWS client from my local term (due to credentials issue); however, they
can probably be changed to use something like what I did with the 
partialmarupload.py (which I think I added to bug 1233615 instead of here).

It would be simpler to have seabld have its own S3 credentials and allow
it to upload.  Currently using my credentials.

Comment 19

[Edmund Wong (:ewong)]

Attachment: [configs] support move to S3. [checked-in]

Due to the fact that this bug was being fixed in conjunction with bug 1233615,
a lot of the fixes weren't ideally placed in the right bug (most of them
were placed in bug 1233615). mea culpa

Comment 20

[Edmund Wong (:ewong)]

Comment on attachment 8736088 [details] [diff] [review]
[configs] support move to S3. [checked-in]

Pushed to buildbot-configs:
https://hg.mozilla.org/build/buildbot-configs/rev/8ddb066be792

Comment 21

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #20)
> Comment on attachment 8736088 [details] [diff] [review]
> [configs] support move to S3. [checked-in]
> 
> Pushed to buildbot-configs:
> https://hg.mozilla.org/build/buildbot-configs/rev/8ddb066be792

Backed out for missing a 2nd patch.

Comment 22

[Edmund Wong (:ewong)]

Attachment: [configs] changes to support the move to S3. [checked-in]

Comment 23

[Edmund Wong (:ewong)]

Comment on attachment 8736090 [details] [diff] [review]
[configs] changes to support the move to S3. [checked-in]

Pushed to buildbot-configs:
https://hg.mozilla.org/build/buildbot-configs/rev/68d9f1b89839

Comment 24

[Edmund Wong (:ewong)]

Attachment: [custom] changes to custom code to support the move to S3 (v1) [checked-in]

Comment 25

[Edmund Wong (:ewong)]

Comment on attachment 8736124 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (v1) [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/8fe55c176d06

Comment 26

[Edmund Wong (:ewong)]

Removing needinfo.

Comment 27

[Edmund Wong (:ewong)]

Attachment: [custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) [checked-in]

Comment 28

[Edmund Wong (:ewong)]

Comment on attachment 8736126 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/958bca875669

Comment 29

[Edmund Wong (:ewong)]

Ftr, I just unset the review flags as Callek wasn't going to be able to
get to them and since they're already in production (and working),
I got Callek's approval for doing this.  Closing this bug.