Closed Bug 1248898 Opened 8 years ago Closed 8 years ago

Please deploy kinto 1.11.2 release to OneCRL STAGE

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX
Milestone:
mozilla47

People

(Reporter: rhubscher, Assigned: dmaher)

References

Details

We just released Kinto 1.11.2, kinto-changes 0.1.0 and kinto-attachment 0.3.0

Release notes:
==============
 - https://github.com/Kinto/kinto/releases/tag/1.11.2
 - https://github.com/mozilla-services/cliquet/releases/tag/2.15.0
 - https://github.com/Kinto/kinto-changes/releases/tag/0.1.0
 - https://github.com/Kinto/kinto-attachment/releases/tag/0.3.0

Comparison:
===========
 - https://github.com/Kinto/kinto/compare/1.8.0...1.11.2
 - https://github.com/mozilla-services/cliquet/compare/2.10.0...2.15.0

Configuration changes:
======================

Please install and configure the kinto-changes module as follow:

    kinto.includes = kinto_changes

    # Kinto changes               
    kinto.event_listeners = changes
    kinto.event_listeners.changes.use = kinto_changes.listener
    kinto.event_listeners.changes.collections = /buckets/blocklists

    kinto.event_listeners.changes.bucket = monitor
    kinto.event_listeners.changes.collection = changes
    kinto.event_listeners.changes.principals = system.Everyone



Please install and configure the kinto-attachment module as follow:

    kinto.includes = kinto_changes kinto_attachment

    # Kinto attachment

    kinto.attachment.base_url = http://cdn.service.org/files/
    kinto.attachment.folder = {bucket_id}/{collection_id}

    kinto.attachment.aws.access_key = <AWS access key>
    kinto.attachment.aws.secret_key = <AWS secret key>
    kinto.attachment.aws.bucket = <bucket name>
    kinto.attachment.aws.acl = <AWS ACL permissions|public-read>

https://kinto.stage.mozaws.net/
Summary: Please deploy kinto 1.10.1 release to kinto STAGE → Please deploy kinto 1.11.2 release to kinto STAGE
Assignee: nobody → dmaher
QA Contact: chartjes
Chris, I'm taking over QA Contact on this bug while we're at the workweek, because we're probably going to deploy this while everybody's here.

I'll keep you cc'ed on the bug, so you can keep track of progress.
QA Contact: chartjes → kthiessen
We created kinto-dist during the work-week to reduce the deployment to a single project and version. The plan is to use that in th efuture for all kinto releases.

It's a python package that pulls all the deps to the right pinned versions.

See https://github.com/Kinto/kinto-dist

I'll leave it up to Ops+QA to decide if we should start with kinto-dist on this push or not, since it involves some extra puppet work I guess.
Target Milestone: --- → mozilla47
I like kinto-dist and am glad to move things in that direction - I'd like to loop in QA for a once-over, however.


On the Jenkins side, thanks to the addition in app.wsgi[0] in said repo, building the deployment artefact should be fairly straightforward.

Of note, kinto-dist introduces the admin panel, which is good, but it also necessitates a mechanism for actually accessing that panel.  I'll work with :bobm and :ulfr (as appropriate) to make the appropriate modifications to the stack.  I suspect this will also implicate a vpn group of some type, but I'm not yet entirely familiar with the ramifications of that statement. :)


[0] https://github.com/Kinto/kinto-dist/commit/5ff3f73a4a80e3d5bcb2953aefaa43078f98761a
Flags: needinfo?(kthiessen)
Chris (:chartjes) and I can thumb-wrestle over who gets to do the QA work here, or we may tandem.

Where is the package going to be deployed for staging?  Did I miss a URL?
Flags: needinfo?(kthiessen)
This bug has become Insufficiently Specific(tm).  The current accepted wisdom is that there won't be a monolithic "Kinto Service" going forward - each service/project will deploy its own Kinto as part of its stack, respectively.

I'm going to go ahead and assume that this bug refers to some *specific instantiation* of Kinto within an existing stack somewhere.  Based on discussions I've had on IRC with :natim and :ulfr today, I am lead to believe that the stack in question is that which serves OneCRL (named, maddeningly, "Kinto" in the svcops world).

If this is true, then:
* kinto-attachment does *not* need to be deployed since OneCRL does not use it.
* kinto-signer *does* need to be deployed (in which case I'll need the config for that).
* cliquet-fxa should be *removed* from the existing stack.

:natim: please confirm, deny, or amplify the above bullet points.  Thank you.
Flags: needinfo?(rhubscher)
phrawzty correct. Maybe the CDN DNS name (https://firefox.settings.services.mozilla.com) should changed as well.
Flags: needinfo?(rhubscher)
Summary: Please deploy kinto 1.11.2 release to kinto STAGE → Please deploy kinto 1.11.2 release to OneCRL STAGE
> The current accepted wisdom is that there won't be a monolithic "Kinto Service" going forward - each service/project will deploy its own Kinto as part of its stack, respectively.

kinto-dist is just a pile of python packages at given stable versions. If you pupettize it, you get packages like kinto-attachment built-in and then you can decide to activate it or not.

I don't think you need to worry about the details of what should be installed in the python side, you should just deploy kinto-dist and then activate the given feature if the project uses it.
Flags: needinfo?(dmaher)
(In reply to Tarek Ziadé (:tarek) from comment #8)=
> I don't think you need to worry about the details of what should be
> installed in the python side, you should just deploy kinto-dist and then
> activate the given feature if the project uses it.

Correct.

The content of comment #0 contains specific details as to the activation of features for a given project, without specifying the project - this is exactly why clarification was required (and a good thing, since the details ended up being incorrect).
Flags: needinfo?(dmaher)
The relevant bits are:

> kinto.includes = kinto_signer.hook
> kinto_signer.resources =
>    source/collection1;destination/collection1
>    source/collection2;destination/collection2
> kinto_signer.autograph.hawk_id = alice
> kinto_signer.autograph.hawk_secret = fs5wgcer9qj819kfptdlp8gm227ewxnzvsuj9ztycsx08hfhzu
> kinto_signer.autograph.server_url = http://localhost:8000
This has been a long, arduous process, full of interesting pitfalls and strange outcomes.  The tl;dr is that there is a fair amount of work on the svcops side that's going to need to happen to get this to a point where we can trust it for prod.  That said, here is where we stand:

* Stage has been deployed using kinto-dist[0].
  * From branches of both svcops ansible[1] and puppet[2].
* DNS has been manually switched such that the service is available at kinto.stage.mozaws.net .
* The service responds to queries; however, all responses are 500 "Internal Server Error".

I will resume work on this deployment on Monday.

[0] 5ff3f73a4a80e3d5bcb2953aefaa43078f98761a
[1] b7da12db05fdd9f9b41fefbcaadea32728aa3338
[2] 8b93f1736a77fb04e2f896ca5c5d7f21049a4dcb
Status: NEW → ASSIGNED
> The tl;dr is that there is a fair amount of work on the svcops side that's going to need to happen to get this to a point where we can trust it for prod.

To be clear, the infrastructure - which is to say the AWS resources - once in place are totally fine.  I am concerned about the resource management lifecycle (and associated processes) as a whole.
Thanks for the report.

I don't know why we're having a 500 here. nginx and process logs might offer more context here.
I also wonder about the name of the deployment (that is, kinto.mozaws.net), since this is OneCRL only, I wonder if we might prefer to change it to onecrl.mozaws.net ?
You can look at the sentry to get an idea of the error: https://sentry.prod.mozaws.net/operations/kinto-stage/group/241579/

I have created the following issue: https://github.com/Kinto/kinto/issues/494
(In reply to Alexis Metaireau (:alexis) from comment #15)
> I also wonder about the name of the deployment (that is, kinto.mozaws.net),
> since this is OneCRL only, I wonder if we might prefer to change it to
> onecrl.mozaws.net ?

While I certainly agree with you in principal, I want to get a single, clean deployment out first before we start messing with anything else. :)
There have been a couple of items discovered thus far during the debugging process:

* The .ini template for Kinto contained within our Puppet repository is legacy, and contains keys whose names were changed in the most recent release.
  * s/maxconn/size/;
* There is an inconsistency in the Stage .ini whereby the permissions backend was configured to use Postgres, but points to a Redis endpoint (this could never have worked). This is configured correctly in the Prod .ini, interestingly.
* The kinto-signer module is _not_ included in kinto-dist and thus cannot be loaded at uwsgi runtime.

I am working with :natim to solve these issues (and any more than may arise).
Pipeline #30 was successfully deployed to Stage with the fixes from comment #18 above. Unfortunately, we're still getting an ISE from the service, which can be traced back to this error from the uwsgi stacktrace:

Mar  8 10:14:16 localhost uwsgi: ValueError: Please specify either kinto.signer.ecdsa.private_key or kinto.signer.ecdsa.public_key in the settings.

Now debugging with :natim.
You need to specify the following values in the configuration file:

> kinto.signer.resources =
>    {source_bucket_id}/{source_collection_id};{destination_bucket_id}/{destination_collection_id}

> kinto.signer.signer_backend = kinto_signer.signer.autograph
> kinto.signer.autograph.hawk_id = {hawk_id}
> kinto.signer.autograph.hawk_secret = {hawk_secret}
> kinto.signer.autograph.server_url = {autograph_server_url}
Well it certainly was a team effort, but we've finally got a stable recipe for pushing Kinto Stage! :)

Pipeline #32
kintoGitRef:	c0687b5e9a62771c5b128471b82f931bf4feff77
SvcopRef:	b7da12db05fdd9f9b41fefbcaadea32728aa3338
PuppetGitRef:	62249f2b833e04257cd1e1b4b2601ab1c6f38ae5

$ http https://kinto.stage.mozaws.net/v1/ | jq '.project_version'
"1.11.2"

$ dig kinto.stage.mozaws.net | grep CNAME
kinto.stage.mozaws.net.	153	IN	CNAME	31-kinto.stage.mozaws.net.



There is another pipeline called "Kinto-Writer" in Jenkins. My guess is that it corresponds with the "write" side of the project - but I'm wary because names can, and often are, deceiving. :/  I'm going to spend some time poking around this pipeline to try and get a better handle on what it is/does.
Kinto-writer has also been deployed to Stage.

Pipeline #13
kintoGitRef:	c0687b5e9a62771c5b128471b82f931bf4feff77
SvcopRef:	d60099ffe4725712899a32e80f7d1ad4fa7b9861
PuppetGitRef:	62249f2b833e04257cd1e1b4b2601ab1c6f38ae5

kinto-writer.stage.mozaws.net



The endpoint in question is part of a restrictive Security Group (as one might expect).  Based on my initial reading of the config for this SG, it looks to be a sort of "minimum viable" group - just enough for the deployment to function, and no more.

For what it's worth, the basic requirements of this bug have been satisfied: Kinto 1.11.2 has been deployed to Stage.  That said, we now need to consider how access to the Write component (aka "Admin") will be implemented - which is outside the scope of *this* bug.
Dan I received this sentry error: https://sentry.prod.mozaws.net/operations/kinto-stage/group/242937/
Are you sure that the Database migration script has been executed?
(In reply to Rémy Hubscher (:natim) from comment #23)
> Dan I received this sentry error:
> https://sentry.prod.mozaws.net/operations/kinto-stage/group/242937/
> Are you sure that the Database migration script has been executed?

Database migration script?  I don't see any references to that earlier in this bug - could you please elaborate?
> That said, we now need to consider how access to the Write component (aka "Admin") will be implemented - which is outside the scope of *this* bug.

I thought this bug is all about the writer side of things as well as the read-part. If you don't want to do that in this bug (which is fine, I don't really care), please open another one and cc us there.

The migration script is what is done when `kinto migrate` is run.
As soon as a new version of kinto is deploy you would need to run ``kinto migrate`` to setup the backends. (permissions, storage and cache)

This needs to be run once.
Blocks: 1255034
I hit a Python error whilst attempting to run the migration - the important part is:

File "/data/kinto/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (cliquet 3.1.0 (/data/kinto/lib/python2.7/site-packages), Requirement.parse('cliquet>=2.15,<3'))

From IRC:

03:33:17 < natim> phrawzty: you are not suppose to use this version of cliquet
                  with that version of kinto. That's a bug in kinto-dist I will
                  fix it
Daniel it should be fixed now, if you want to try a new update.
(In reply to Rémy Hubscher (:natim) from comment #28)
> Daniel it should be fixed now, if you want to try a new update.

As of Thu 10 Mar 2016 15:42:49 UTC the master branch of the kinto-dist repo has not been updated. Has the required fix been implemented in an upstream library? Please confirm.
>  Has the required fix been implemented in an upstream library?

Yes it didn't need a kinto-dist change.
Superseeded by Bug 1255776
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Blocks: 1255776
No longer blocks: 1255034
You need to log in before you can comment on or make changes to this bug.