Closed Bug 654148 Opened 13 years ago Closed 13 years ago

Deploy Python Reg and Sreg

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: telliott, Assigned: petef)

References

Details

RPMs to be deployed:

server-core rpm-1.0-4
server-sreg rpm-0.1-1
server-reg rpm-0.1-14
account-portal rpm rpm-1.0-1

Changes:

Replaces reg and sreg with the python versions. 
Updates LDAP to work with both the current system and new proposed structure.
Miscellaneous fixes

Notable bugs:

Bug 648079: Putting CEF signatures back into account portal
Bug 636514: New sreg API
Bug 636514: Better handling of password resets requested with no email
Bug 650055: Deleting a user now done through sreg
Bug 652169: Removing proxy calls from reg (they're now done through the auth library)

Notably not present in this release: changing your email address does not, at this time, change your username.

QA plan

Because this push changes a central feature of the core library, and completely replaces reg and sreg, all features of account-portal and the reg API will need to be exercised. 

For account portal, this involves test most available functions, including login, forgot password flow, changing password and deleting an account (the sync settings are unchanged and do not need testing)

For the API the spec is at https://wiki.mozilla.org/Services/Sync/Server/API/User/1.0. The important things to test are creating a user and getting a node. It would be good if we can also test email changes, password changes and deleting an account through the API as well.

Security Documentation

The bulk of this code has already been reviewed, and the infratstructure setup is, with one exception, unchanged. It would be good for someone to give a quick eyeball to the sreg code that handles user deletion, as that is a new function on sreg.
I'll work on getting this in staging.
Assignee: nobody → petef
Status: NEW → ASSIGNED
Now on the wiki page for editing at https://intranet.mozilla.org/Services/User/Releases/0.1
new server-sreg tag: rpm-0.2-1
rpm-0.2-2 now.
the current list:

server-core rpm-1.0-6
server-sreg rpm-0.2-3
server-reg rpm-0.1-14
account-portal rpm rpm-1.0-1
OK. toby & i are happy with how staging is working right now, ready for QA handoff.
Assignee: petef → nobody
Status: ASSIGNED → NEW
Whiteboard: [needs qa]
Depends on: 655268
Depends on: 655285
Depends on: 655282
This has been pushed back a week due to a combination of ACL issues and a
Depends on: 655281
... missed piece of functionality hidden by them. New tags should be up once we address the final issues.
New RPMs for this week:

server-core rpm-1.2-1
server-sreg rpm-0.4-1
server-reg rpm-0.2-1
account-portal rpm-1.0-1
Assignee: nobody → petef
(In reply to comment #9)
> server-core rpm-1.2-1
> server-sreg rpm-0.4-1
> server-reg rpm-0.2-1
> account-portal rpm-1.0-1

deployed in stage, ready for verification & handoff to QA.
Adding one more blocking bug for this deployment to be ready to go.
Depends on: 656767
Revised account-portal tag: rpm-1.1-1

All others should be unchanged.
Build problems:

$ make build build_rpms SERVER_CORE=rpm-1.2-1 ACCOUNT_PORTAL=rpm-1.1-1
[...]

+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd AccountPortal-0.1
/var/tmp/rpm-tmp.37173: line 31: cd: AccountPortal-0.1: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.37173 (%prep)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.37173 (%prep)
error: command 'rpmbuild' failed with exit status 1
make: *** [build_rpms] Error 1
new ap tag: rpm-1.1-2
stage registration server and account portal look good.
Whiteboard: [needs qa]
Pete can you write down here the versions for *all* deployed RPMs so we can pin our Makefiles, thx !
In general, we guarantee honoring spec minversion requirements (version >= N.NN-N), but may push newer versions of packages as necessary to resolve system and/or security issues.

RPM dependency versions are not currently pinned across multiple servers, even within the same site.  There is no one canonical version for, say, perl-DBD-MySQL, or any of the other ancillary dependencies that might come along unexpectedly, even within RHEL5.

We do allow yum to upgrade packages with a minversion requirement in an application rpm (eg. syncstorage) requires a version newer than is installed on the system, as long as the newer version is available in the repository.
We still need Pete to give us the list of RPM versions he generated out of the Makefile. Not the indirect dependencies, but the RPM built out of the PyPI packages like WebOb etc..

This is very important so the developer works with the same versions that stage.

That does not mean we cannot update them, on the contrary, Ops or Dev can raise the versions before the next push. That just means that we're able to run the same server when a bug must be reproduced. And also that the Ops that generates the RPM do generate the same app that the developer built and tested.
These are from the latest RPM build on stage for reg:

python26-cef-0.2-1.noarch.rpm
python26-mako-0.4.1-1.noarch.rpm
python26-markupsafe-0.12-1.x86_64.rpm
python26-memcached-1.47-1.noarch.rpm
python26-mysql-python-1.2.3-1.x86_64.rpm
python26-paste-1.7.5.1-1.noarch.rpm
python26-pastedeploy-1.3.4-1.noarch.rpm
python26-pastescript-1.7.3-1.noarch.rpm
python26-recaptcha-client-1.0.6-1.noarch.rpm
python26-routes-1.12.3-1.noarch.rpm
python26-simplejson-2.1.6-1.x86_64.rpm
python26-sqlalchemy-0.6.6-1.noarch.rpm
python26-webob-1.0.7-1.noarch.rpm
python26-wsgiproxy-0.2.2-1.noarch.rpm
- reg pinned: https://hg.mozilla.org/services/server-reg/rev/dad90a485138
- sreg pinned: https://hg.mozilla.org/services/server-sreg/rev/d7274c57f490

You can now use the server-reg or server-sreg repository to create a collection of RPMs for each one of these servers, independently from server-full --and like you used to do there--

The RPMS are also tried in a mach environment on centos5:

- http://tarek1.svc.mtv1.mozilla.com/view/Reg/
- http://tarek1.svc.mtv1.mozilla.com/view/SReg/

The next steps are for me to make them build in rhel6 as well, and to copy over the functional tests from server-full that are run against stage, to be run against a single sreg or reg server where it makes sense.

Also, there's a simple "import app; app.run" call for now in the chroot, but we could add more tests in there.

Last: I am also pinning account portal, so I make the assumption that you currently have the same versions of the packages, expect for sqlalchemy which was pinned to 0.6.7. 

Can you let me know the versions deployed for:

- pycryptopp
- pycrypto
- wsgi_intercept (Is this needed in production, Toby?)
- BeautifulSoup
(In reply to comment #20)
> Last: I am also pinning account portal, so I make the assumption that you
> currently have the same versions of the packages, expect for sqlalchemy
> which was pinned to 0.6.7. 

correct

> Can you let me know the versions deployed for:
> 
> - pycryptopp
> - pycrypto
> - wsgi_intercept (Is this needed in production, Toby?)
> - BeautifulSoup

python26-pycryptopp-0.5.29-1
python26-pycrypto-2.3-1
python26-wsgi_intercept-0.4-1
python26-beautifulsoup-3.2.0-1
(In reply to comment #20)

> - wsgi_intercept (Is this needed in production, Toby?)

I believe this is just for testing.
Depends on: 621283
So, as told to QA, the test plan needs now to check those:

- registering a user with a containing a non-ascii char, like é.
- change a user password to something containing a non-ascii char, like é.
New tags have been made:

- python-services-1.3-1
- python-sreg-0.5-1
- python-reg-0.3-1

python-accountportal remains unchanged (1.1-2)

I am working with atoll to deploy those on the new dev box setup (centos5/rhel6)
I've deployed those dev servers:

-ap1 (update)
-ap2 (update)
-reg1
-reg2
-sreg1
-sreg2

Richard, Pete, fixes to do to puppet:

reg1 and reg2 => fixed /etc/nginx/conf.d/syncreg.conf so it uses tcp:8000, not a sock file (gunicorn was using tcp:8000)

same thing for sreg1/sreg2 in /etc/nginx/conf.d/syncsreg.conf

Everything else looks good, so I guess Toby can do a quick health check on AP and we can deploy on stage for QA to start.
Adding Bug 625230 since this impacts Infrasec
Depends on: 625230
The tags in comment #24 are the desired tags for this week's staging and next Monday's train release.  To clear up any confusion, the changes listed above in comment 25 are linked to the dev environment and do not impact any stage or prod deployments.
tags deployed to stage:

SERVER_CORE=1.3-1
SERVER_REG=0.3-1
SERVER_SREG=0.5-1
ACCOUNT_PORTAL=1.1-2

handed off to QA for testing.
latest tags:

SERVER_CORE=1.5-1
SERVER_REG=0.4-1
SERVER_SREG=0.5-1
ACCOUNT_PORTAL=1.1-2

although there is a bug with deleting account/clearing sync data from AP w/unicode passwords (linked bug coming soon).
QA signs off testing on staging
SERVER_CORE=1.6-1
SERVER_REG=0.5-1
SERVER_SREG=0.5-1
ACCOUNT_PORTAL=1.1-2
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
verified against production
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.