688309 - Use only master in mozillians-dev and mozillians.allizom

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Austin King [:ozten]]

Please change the consumer/provider settings to both point to our slapd *master*.

This will allow us to troubleshoot several bugs that are not reproducible in our local development environments.

thanks!

Comment 1

[Austin King [:ozten]]

To make this happen, just change this:

    LDAP_SYNC_CONSUMER_URI = LDAP_SYNC_PROVIDER_URI

Comment 2

[Justin Dow [:jabba]]

Can you reproduce by running two local slapd's on your local env? The reason I'm hesitant to turn this back off is that if something isn't working because of replication, it needs to be known now, since prod will be using a master/slave setup. If we remove the master/slave setup from dev and stage, then we won't be able to go to prod with a master/slave setup.

Comment 3

[Austin King [:ozten]]

I'm proposing we switch this off for our 1.0 prod release.

The more I dig, the more I don't think this is ready for release on the code and monitoring side.

Comment 4

[Justin Dow [:jabba]]

So what happens when it launches and hundreds to thousands of people sign up and it kills our only server? Then we have to take the site back down, fix the code, qa and re-launch. Isn't it better to fix it ahead of time? I understand there are deadlines, but shipping a broken product that causes both IT and Devs a lot of pain around the launch can't be better than shipping an awesome product a little late? We have certain SLAs, and in order to be able to provide those SLAs, the products we back with the SLA needs to be scalable. This means >1 server needs to work.

We ran into this exact situation with the markup project because the site could only handle connections from a single webhead. It was embarrassing for us to launch it, have it fail spectacularly and have to pull it back down and rush to fix it. It was the same situation where product owners put pressure on us to ship it prematurely. I don't want a repeat of that.

I won't make the call either way on this, but my advice is to get mutliple LDAP backends supported so that we can scale this as needed both for redundancy and for performance.

Comment 5

[Corey Shields [:cshields]]

(In reply to Austin King [:ozten] from comment #3)
> I'm proposing we switch this off for our 1.0 prod release.
> 
> The more I dig, the more I don't think this is ready for release on the code
> and monitoring side.

No, we require redundancy to go live with this..

Please fix the underlying problems..

Comment 6

[Fred Wenzel [:wenzel]]

If it helps you debug, I am sure IT will gladly change the setting on dev, but I agree that if it doesn't work with replication, it isn't be ready for prime time.

Have you narrowed it down somewhat? If you need a second set of eyes let me know.

Are we using one instance to read, one to write? If so, we might be experiencing a replication lag resulting in the read instance not being up to date immediately thereafter?

Comment 7

[Austin King [:ozten]]

(In reply to Corey Shields [:cshields] from comment #5)

There are two underlying problems.

1) Read only views hitting ro slaves which are out of sync.

We have no monitoring in place on stage to see current replication lag. 

What can we expect for an SLA on this lag? Will master and slave be connected via gigabit ethernet, etc? I think I'm seeing lag in stage, which is surprising given it's not under heavy writes.

I've filed Bug#688536 to add monitoring.

2) LDAP pooling and consumer vs provider codebase not ready for prime-time.

Turning this off as requested in Comment#0 for 24 hours would be a great asset in troubleshooting the stack traces we're getting. Per Comment#2, it sounds like this is on me to setup a env with lag...

I don't feel comfortable with our current codebase in a master/slave configuration for 1.0. We don't handle replication lag well, AFAICT.

I'll circle back with aakash and David to let them know.

> So what happens when it launches and hundreds to thousands of people sign up and it kills our only server? 

I think this has to scale to 30K casual users.

> We ran into this exact situation with the markup project

Markup was linked from snippets and is an Engagement campaign. This is a community tool.

I totally agree that we shouldn't ship a broken product.

I'm not code complete if we're shipping with master/slave. I'm happy to rewrite these pieces, but we'll want monitoring in place also.

Comment 8

[Austin King [:ozten]]

(In reply to Fred Wenzel [:wenzel] from comment #6)

> If it helps you debug, I am sure IT will gladly change the setting on dev, but I agree that 
> if it doesn't work with replication, it isn't be ready for prime time.

> Have you narrowed it down somewhat? If you need a second set of eyes let me know.

There are intermittent issues in stage that don't appear in -dev. They don't repro in local development environments.

> Are we using one instance to read, one to write? If so, we might be experiencing a 
> replication lag resulting in the read instance not being up to date immediately thereafter?

Yes, this was brought up yesterday in email and today in the status call.

Comment 9

[Jason Thomas [:jason]]

I have configured -dev to use Master only.

Comment 10

[Austin King [:ozten]]

Filed Bug#688612 to get django-auth-ldap to use master.

Working through requirements in http://etherpad.mozilla.com:9000/mozillians-ldap-pooling-and-replication-design

Closing won't fix.