Closed Bug 1310041 Opened 8 years ago Closed 7 years ago

Please create test S3 bucket on AWS for testing attachment migration from bugzilla-dev.allizom.org

Categories

(bugzilla.mozilla.org :: Infrastructure, defect)

Product:
bugzilla.mozilla.org
Component:
Infrastructure
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: dkl, Assigned: fubar)

References

Details

I would like a specific namespace under our BMO S3 account that was created that we can use for a test migration from bugzilla-dev before the production migration on November 19th. Please let me know if I can do anything to help with this.

dkl
NI on :gozer, since it requires creating IAM keys which I don't know that I can do (although if I can and it's easy to convey, let me know!).
Flags: needinfo?(gozer)
Could we use the existing attachment buckets in stage for this purpose, AFAIK, it's not being used for anything
Flags: needinfo?(gozer)
s3://bugzilla-stage-attachments-ab7ac749-afcf-412b-9f66-20323297405e/
Credentials sent to :fubar via GPG-email

Subject: Bug 1310041
Message-ID: <442ed149-2108-5141-8090-187934f26361@mozilla.com>
(In reply to Philippe M. Chiasson (:gozer) from comment #2)
> Could we use the existing attachment buckets in stage for this purpose,
> AFAIK, it's not being used for anything

Except we're going to want SCL3 staging to have an S3 bucket to use.
(In reply to Philippe M. Chiasson (:gozer) from comment #4)
> Credentials sent to :fubar via GPG-email
> 
> Subject: Bug 1310041
> Message-ID: <442ed149-2108-5141-8090-187934f26361@mozilla.com>

fubar, if you can add the credentials for the S3 bucket to data/params on bugzilla-dev.allizom.org, I can run the test migration with the attachments we have on the system. 

thanks
dkl
Flags: needinfo?(klibby)
NI on gozer; I need an answer to my concern in #c5. Mixing SCL3 dev and Nubis stage seems like a recipe for disaster. If it's a problem setting it up in Nubis, then I'll add it to the devservices account.
Flags: needinfo?(klibby) → needinfo?(gozer)
(In reply to Kendall Libby [:fubar] from comment #7)
> NI on gozer; I need an answer to my concern in #c5. Mixing SCL3 dev and
> Nubis stage seems like a recipe for disaster.

Good point, actually, I thought this bucket was to be used for some sort of one-time
deal, so figured we could just *borrow* the stage one, but that's clearly not the case.

> If it's a problem setting it
> up in Nubis, then I'll add it to the devservices account.

Would that be easy for you to do?

Right now, Nubis has been architected with support for prod/stage, but no provisions have
yet been made to support dev, so can't easily achieve what you'd want here.

I recommend you go the devservices road for now.

On the topic, has there ever been any sort of planning for running bugzilla-dev in AWS at some point?

NI on r2 on that subject
Flags: needinfo?(gozer) → needinfo?(riweiss)
There we never received a requirement for a bugzilla-dev in AWS. All designs, which were reviewed with the bmo team, included just a stage and a prod. Is there going to be an ongoing need for a dev environment, or is this a one-time thing?
Flags: needinfo?(riweiss)
(In reply to Philippe M. Chiasson (:gozer) from comment #8)
> > If it's a problem setting it
> > up in Nubis, then I'll add it to the devservices account.

Care to email me details of ACLs/policies you're using on the stage/prod buckets so I can make dev match, or at least be close enough?

> On the topic, has there ever been any sort of planning for running
> bugzilla-dev in AWS at some point?

(In reply to Richard Weiss [:r2] from comment #9)
> There we never received a requirement for a bugzilla-dev in AWS. All
> designs, which were reviewed with the bmo team, included just a stage and a
> prod. Is there going to be an ongoing need for a dev environment, or is this
> a one-time thing?

This should be part of the conversation about BMO living full-time in AWS.
Flags: needinfo?(gozer)
Agreed.
(In reply to Kendall Libby [:fubar] from comment #10)
> (In reply to Philippe M. Chiasson (:gozer) from comment #8)
> > > If it's a problem setting it
> > > up in Nubis, then I'll add it to the devservices account.
> 
> Care to email me details of ACLs/policies you're using on the stage/prod
> buckets so I can make dev match, or at least be close enough?

Nothing to it, really.

One bucket, acl private

One IAM role/user with a policy that grants s3:* to that user to that bucket.

Policy looks like:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bugzilla-prod-attachments-XXX/*",
        "arn:aws:s3:::bugzilla-prod-attachments-XXX"
      ],
      "Effect": "Allow"
    }
  ]
}
Flags: needinfo?(gozer)
fubar, can we get the dev aws credentials installed on bugzilla-dev.allizom.org and bugzilla.allizom.org? I want to manually run the migration tests on those today/tomorrow. Also we can go ahead and put the proper keys on production as well. I would like to run the production migration from the admin node on Friday so to save time during the TCW.

Thanks
dkl
Flags: needinfo?(klibby)
Added to the admin node in root's homedir, so you'll need to use sudo to access them. Each environment has their own credentials; with the AWS CLI use the --profile option to specify which to use - dev_att, stage_att, or prod_att.

Also added to data/params for dev and stage, along with the URL for the bucket (although I'm not certain the stage bucket is in us-east-1!), and pushed out to the web heads.

Let me know that they both work and I'll add prod.
Flags: needinfo?(klibby) → needinfo?(dkl)
This needs /etc/environment set up on all of the nodes to allow them access to S3 via the DC proxies; working on that now.
Assignee: nobody → klibby
Dev and stage are done, so you can proceed with testing there; working on prod now.
All prod hosts are done; when we've tested dev/stage, let me know and I'll add and push the creds for prod.
(In reply to Kendall Libby [:fubar] from comment #17)
> All prod hosts are done; when we've tested dev/stage, let me know and I'll
> add and push the creds for prod.

I am getting an error on bugzilla-dev that the bucket is invalid. Stage is working fine and I am running the migration now.

--error--

root@web5.stage.bugs.scl3 bugzilla-dev.allizom.org]# perl scripts/migrate-attachments.pl --copy database s3
Copy 603 attachments from database to s3?

Press <Ctrl-C> to stop or <Enter> to continue..

.Failed to add attachment 8591929 [details] [review] to S3: The specified bucket does not exist
Failed to add attachment ID 8591929 to S3: The specified bucket does
not exist

's3_bucket' => 'https://s3.amazonaws.com/moz-bugzilladev-attach/',

dkl
Flags: needinfo?(dkl) → needinfo?(klibby)
There's something different in the setup of the stage/prod buckets that I can't seem to match with dev. If we specify the bare bucket name for dev, it works, though:

web5.stage.bugs.scl3#perl scripts/migrate-attachments.pl --copy database s3
Copy 604 attachments from database to s3?

Press <Ctrl-C> to stop or <Enter> to continue..

............................................................60/604 (9%)
............................................................120/604 (19%)
............................................................180/604 (29%)
............................................................240/604 (39%)
............................................................300/604 (49%)
............................................................360/604 (59%)
............................................................420/604 (69%)
............................................................480/604 (79%)
............................................................540/604 (89%)
............................................................600/604 (99%)
....604/604 (100%)

Attachments stored: 0


I'll note that attempting to access stage/prod via un-auth'ed curl, I get a redirect to an endpoint that looks like it's set up for static web hosting in s3:

<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>bugzilla-stage-attachments</Bucket><Endpoint>bugzilla-stage-attachments.s3.amazonaws.com</Endpoint>

But I tried enabling that on dev and it didn't work. Since dev will always be slightly different, I'm willing to just go with the bare bucket name. We may need to rely a little more on staging to catch anything odd before it hits prod, though.
Flags: needinfo?(klibby)
bugzilla-dev.allizom.org has been switched to S3 as primary and I have done some preliminary testing which has been so far successful. I will do some testing on bugzilla.allizom.org once it finishes migrating. Taking a very long time. For production, the plan is to try and utilize the Nubis cluster to do the migrations as going from EC2 to S3 in the same region should be much faster.

dkl
dkl tells me this is fine now.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.