493979 - Personas uplift / Capacity and performance planning

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Toby Elliott [:telliott]]

We're starting to show the seams on the sm-personas box - it's not really equipped for handling a product of this magnitude. The getpersonas.com url still handles the majority of the load, but the underlying problem is fourfold:

1) Disk space: This has been partially alleviated by the mounting of an additional disk, on which I'm logging and doing some archiving. Even so, we're still at 70% on the disk, and it's always going to need monitoring.

2) Performance: The CPU allocation is pushing it, and often hits 100%. Because of the simplicity of the system, external perfomance appears OK for now, but it's going to get worse and there are occasional hiccups.

3) Redundancy: The db is backed up by me doing a periodic copy. Log removal is ad hoc.

4) Capabilities: Many of the dynamic functions we want to do - search, administration, etc - are only possible on this system, and not the static one. This has a twofold effect of slowing development (since some features, such as search, just can't launch) and encouraging traffic to the dynamic box, compounding the other problems.

Comment 1

[Suneel]

a complete solution to this shouldn't be a blocker to 1.4. moving this to 1.5 and setting dependencies for 489246 and 493876 (both mapped to 1.5) on this fix.

Comment 2

[matthew zeier [:mrz]]

Should we start looking at transitioning this to a hybrid Labs prod enviro where IT handles scaling app & db ?

Comment 3

[Toby Elliott [:telliott]]

Yes. If we get a quick hire for the labs-it guy, this would be a great introductory project for them. If not, you and I can coordinate to put something together.

Comment 4

[matthew zeier [:mrz]]

When you give up on waiting for Labs IT, flip it over to server-ops and we'll take over.

Comment 5

[Justin Fitzhugh]

dropping my support in this - would be good to see this uplifted on both the webdev side and ops...

Comment 6

[matthew zeier [:mrz]]

Moving this over to server-ops.  

Since Toby was proactive on this we have some time to plan this.  We'll shoot to touch bases after Toby's back from vacation.

Comment 7

[matthew zeier [:mrz]]

Meant to give this to oremj earlier, clicked the wrong buttons or something.

Comment 8

[Jeremy Orem [:oremj]]

Is there a plan for this written up anywhere?

Comment 9

[Suneel]

Plan is not written up. Next step is for mrz, Toby, and Suneel to determine next steps. One of us will write these up and post.

Comment 10

[matthew zeier [:mrz]]

Mostly waiting for Toby to get back, so next week or so.

Comment 11

[Toby Elliott [:telliott]]

Sat down with Zandr today to go over what we need. He'll build us a LAMP stack in production (and will no doubt update this bug as he progresses). We'll then push a copy of the site live to that set of machines.

I think we can bring down personas.services for a couple minutes to move the database over to the new setup. If we punch a hole in the firewall, we can then have personas.services do subsequent reads and writes to the new db, which will prevent any weirdness during the CNAME change. Once we're happy with the new site, we move the getpersonas CNAME over, and redirect personas.services.mozilla.com to getpersonas and we should be all set.

Note that we'll need to change any url references to personas.services, though the redirect will make them work properly anyway.

Comment 12

[Toby Elliott [:telliott]]

An update: personas is now working off the new db.

Additionally, we have the site mostly live: http://pm-app-personas01/

The images have only been synched across once - Alexandr is putting a cron to do this regularly. But that should be enough to start testing.

If we open up the beta, I would recommend encouraging users to not upload through the beta site, as that will be a bit of a pain for us to administer (targetted uploads to test things should be fine).

Comment 13

[Toby Elliott [:telliott]]

Further update:

Memcache is in place and appears to be working. We'll need to test that carefully.

Stephen has done some basic testing and things look pretty good. Ryan is working on making favorites look good.

Zandr is getting us an external address and setting up the rsync

I think we can release this for beta testing on Monday. Once we do, we should redirect /upload to personas.services so that we keep all the images flowing into one location, but our users can try out the rest of the functionality on the new site.

Comment 14

[Zandr Milewski [:zandr]]

We are almost ready to launch and were actually live briefly on Weds. evening. We're waiting for an SSL cert for getpersonas.com. (#501845) and plan to launch Monday.

Comment 15

[Zandr Milewski [:zandr]]

Uplift complete. Share and Enjoy

http://www.getpersonas.com/