Tracking bug for migration from ftp.m.o to S3 for SeaMonkey

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: ewong, Assigned: ewong)

Tracking

Trunk
Dependency tree / graph

Firefox Tracking Flags

(firefox45 affected)

Details

Attachments

(8 attachments, 5 obsolete attachments)

3.17 KB, patch
Details | Diff | Splinter Review
3.83 KB, patch
Details | Diff | Splinter Review
3.28 KB, patch
Details | Diff | Splinter Review
5.92 KB, patch
Details | Diff | Splinter Review
1.88 KB, patch
Details | Diff | Splinter Review
3.40 KB, patch
Details | Diff | Splinter Review
8.69 KB, patch
Details | Diff | Splinter Review
6.24 KB, patch
Details | Diff | Splinter Review
(Assignee)

Description

3 years ago
We're migrating off ftp.mo to S3 (pending approval, just getting this bug
up) on the 17th November, so there's a bunch of stuff that needs to be done. 

+++ This bug was initially created as a clone of Bug #1213721 +++

This is the get-off-the-netapp move from ftp cluster in SCL3, moving onto AWS S3. Cloud Ops is driving this, we need to adjust where we upload things.

We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on Oct 13.

Bug 1186297 has been tracking the CI side, but I'll add other bugs here.
Bug 1165405 has all the bugs for release automation fixes.
(Assignee)

Updated

3 years ago
Severity: normal → blocker
(In reply to Edmund Wong (:ewong) from comment #0)
> We're migrating off ftp.mo [...]

Does this mean that the download URL for trunk and aurora nightlies will change, or will they be redirected or redefined to wherever the nightlies will henceforward be published?

> We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on
> Oct 13.
[...]

I suppose you mean November 13 (today), not 11 months from now?
(Assignee)

Comment 2

3 years ago
(In reply to Tony Mechelynck [:tonymec] from comment #1)
> (In reply to Edmund Wong (:ewong) from comment #0)
> > We're migrating off ftp.mo [...]
> 
> Does this mean that the download URL for trunk and aurora nightlies will
> change, or will they be redirected or redefined to wherever the nightlies
> will henceforward be published?
> 

Downloading will most likely be from archive.mozilla.org (though I'm not
100% sure on this).  This mostly will affect how the releng side uploads
the releases/tinderbox stuff because upload and downloading 'stuff' will
be from different hosts. 

> > We are going to migrate firefox/, mobile/, thunderbird/ and seamonkey/ on
> > Oct 13.
> [...]
> 
> I suppose you mean November 13 (today), not 11 months from now?

No,  that was the original scheduled date I believe.  It's now due to be
moved on Nov 17, which is three days from now.
(Assignee)

Updated

3 years ago
Assignee: nobody → ewong
Status: NEW → ASSIGNED
(Assignee)

Updated

3 years ago
Depends on: 1212259
(Assignee)

Comment 3

3 years ago
Created attachment 8687604 [details] [diff] [review]
[configs] proposed wip patch (v1)
Attachment #8687604 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 4

3 years ago
Due to the differences between SeaMonkey's releng and Firefox's releng, 
copying Firefox's changes to our buildbotcustom code isn't the right
approach.

Wouldn't it be easier if we just find out the needed urls (which I think
my buildbot-config changes cover), the needed credentials and the
needed procedures to enable our releng process to upload to the staging?
Right now, Firefox uses mozharness to do the general release stuff,
specifically the push-candidate-to-releases.py.  

Callek, any ideas how I should proceed?
Flags: needinfo?(bugspam.Callek)
(Assignee)

Comment 5

3 years ago
Created attachment 8687754 [details] [diff] [review]
[configs] proposed patch (v2)
Attachment #8687604 - Attachment is obsolete: true
Attachment #8687604 - Flags: review?(bugspam.Callek)
Attachment #8687754 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 6

3 years ago
Created attachment 8690531 [details] [diff] [review]
[tools] change paths to reflect the ftp.mo->S3 changes. [checked-in]

[going for a post-land-review] as it will help with the manually patched
configs repo using the configs patch.
Attachment #8690531 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 7

3 years ago
Comment on attachment 8690531 [details] [diff] [review]
[tools] change paths to reflect the ftp.mo->S3 changes. [checked-in]

Pushed to tools repo:
 https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/164592942e31
(Assignee)

Updated

3 years ago
Attachment #8690531 - Attachment description: [tools] change paths to reflect the ftp.mo->S3 changes. → [tools] change paths to reflect the ftp.mo->S3 changes. [checked-in]
(Assignee)

Comment 8

3 years ago
Created attachment 8690647 [details] [diff] [review]
[puppet] add upload host to knownhosts template. (v1)[checked-in]

for a post-land-review
Attachment #8690647 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 9

3 years ago
Comment on attachment 8690647 [details] [diff] [review]
[puppet] add upload host to knownhosts template. (v1)[checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/e1d0fff5265a
(Assignee)

Updated

3 years ago
Attachment #8690647 - Attachment description: [puppet] add upload host to knownhosts template. (v1) → [puppet] add upload host to knownhosts template. (v1)[checked-in]
(Assignee)

Comment 10

3 years ago
Created attachment 8691666 [details] [diff] [review]
[tools] Fixed upload paths in post_upload.py [checked-in]
Attachment #8691666 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 11

3 years ago
Comment on attachment 8691666 [details] [diff] [review]
[tools] Fixed upload paths in post_upload.py [checked-in]

Pushed to tools repo:
https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/752df09c4a82

for post-land-review
Attachment #8691666 - Attachment description: [tools] Fixed upload paths in post_upload.py → [tools] Fixed upload paths in post_upload.py [checked-in]
(Assignee)

Comment 12

3 years ago
Created attachment 8695777 [details] [diff] [review]
[puppet] setup s3 credentials (v1)
Attachment #8695777 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

3 years ago
Blocks: 1230393

Updated

3 years ago
Blocks: 1233615

Updated

3 years ago
Depends on: 1213721
(Assignee)

Comment 13

3 years ago
Created attachment 8728273 [details] [diff] [review]
[puppet] support s3 credentials.
Attachment #8695777 - Attachment is obsolete: true
Attachment #8695777 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 14

3 years ago
Created attachment 8728274 [details] [diff] [review]
[puppet] support s3 credentials. (v3) [checked-in]
Attachment #8728273 - Attachment is obsolete: true
(Assignee)

Comment 15

3 years ago
Comment on attachment 8728274 [details] [diff] [review]
[puppet] support s3 credentials. (v3) [checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/416b6a95f4d6
Attachment #8728274 - Attachment description: [puppet] support s3 credentials. (v3) → [puppet] support s3 credentials. (v3) [checked-in]
(Assignee)

Comment 16

3 years ago
Created attachment 8728278 [details] [diff] [review]
[puppet] support s3 credentials fix (v1) [checked-in]
(Assignee)

Comment 17

3 years ago
Comment on attachment 8728278 [details] [diff] [review]
[puppet] support s3 credentials fix (v1) [checked-in]

Pushed to puppet:
https://hg.mozilla.org/SeaMonkey/puppet/rev/2f33a204fe67
Attachment #8728278 - Attachment description: [puppet] support s3 credentials fix (v1) → [puppet] support s3 credentials fix (v1) [checked-in]
(Assignee)

Comment 18

2 years ago
the scripts which were located on stage.mo have been 'translated' to using
the AWS client from my local term (due to credentials issue); however, they
can probably be changed to use something like what I did with the 
partialmarupload.py (which I think I added to bug 1233615 instead of here).

It would be simpler to have seabld have its own S3 credentials and allow
it to upload.  Currently using my credentials.
(Assignee)

Comment 19

2 years ago
Created attachment 8736088 [details] [diff] [review]
[configs] support move to S3. [checked-in]

Due to the fact that this bug was being fixed in conjunction with bug 1233615,
a lot of the fixes weren't ideally placed in the right bug (most of them
were placed in bug 1233615). mea culpa
Attachment #8687754 - Attachment is obsolete: true
Attachment #8687754 - Flags: review?(bugspam.Callek)
Attachment #8736088 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 20

2 years ago
Comment on attachment 8736088 [details] [diff] [review]
[configs] support move to S3. [checked-in]

Pushed to buildbot-configs:
https://hg.mozilla.org/build/buildbot-configs/rev/8ddb066be792
Attachment #8736088 - Attachment description: [configs] support move to S3. → [configs] support move to S3. [checked-in]
(Assignee)

Comment 21

2 years ago
(In reply to Edmund Wong (:ewong) from comment #20)
> Comment on attachment 8736088 [details] [diff] [review]
> [configs] support move to S3. [checked-in]
> 
> Pushed to buildbot-configs:
> https://hg.mozilla.org/build/buildbot-configs/rev/8ddb066be792

Backed out for missing a 2nd patch.
(Assignee)

Comment 22

2 years ago
Created attachment 8736090 [details] [diff] [review]
[configs] changes to support the move to S3. [checked-in]
Attachment #8736088 - Attachment is obsolete: true
Attachment #8736088 - Flags: review?(bugspam.Callek)
Attachment #8736090 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8736090 - Attachment description: [configs] changes to support the move to S3. → [configs] changes to support the move to S3. [checked-in]
(Assignee)

Comment 23

2 years ago
Comment on attachment 8736090 [details] [diff] [review]
[configs] changes to support the move to S3. [checked-in]

Pushed to buildbot-configs:
https://hg.mozilla.org/build/buildbot-configs/rev/68d9f1b89839
(Assignee)

Comment 24

2 years ago
Created attachment 8736124 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (v1) [checked-in]
Attachment #8736124 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 25

2 years ago
Comment on attachment 8736124 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (v1) [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/8fe55c176d06
Attachment #8736124 - Attachment description: [custom] changes to custom code to support the move to S3 (v1) → [custom] changes to custom code to support the move to S3 (v1) [checked-in]
(Assignee)

Comment 26

2 years ago
Removing needinfo.
Flags: needinfo?(bugspam.Callek)
(Assignee)

Comment 27

2 years ago
Created attachment 8736126 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) [checked-in]
Attachment #8736126 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 28

2 years ago
Comment on attachment 8736126 [details] [diff] [review]
[custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/958bca875669
Attachment #8736126 - Attachment description: [custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) → [custom] changes to custom code to support the move to S3 (remaining paths changes)(v1) [checked-in]
(Assignee)

Updated

2 years ago
Attachment #8690531 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8690647 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8691666 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8736090 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8736124 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8736126 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 29

2 years ago
Ftr, I just unset the review flags as Callek wasn't going to be able to
get to them and since they're already in production (and working),
I got Callek's approval for doing this.  Closing this bug.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.