611582 - Improve performance of BzAPI machine (bugzilla-api01.sj.mozilla.com)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Gervase Markham [:gerv]]

Attachment: Weekly memory graph from munin for api-dev

The BzAPI is being used by quite a few services now, and is coming under increasing load. At zandr's suggestion, I installed Munin a couple of days ago, and my unskilled analysis suggests that the memory is maxed out. Can the VM be allocated some more, please? Or, if that's not the problem, can something else be done?

Any sysadmin with access to the bzapi machine can do:

ssh [e-mail] -L 5050:localhost:80

and find the Munin info on http://localhost/munin/.

I'll attach the memory graph. The discontinuities are reboots.

Gerv

Comment 1

[matthew zeier [:mrz]]

You have 2GB allocated - how much do you suspect you need?  And when can this be done?

At what point can this app be handed off to my team to manage?

Comment 2

[Gervase Markham [:gerv]]

mrz: I'm not the expert; I was hoping your team could look at my graphs and tell me how much. But bumping it to 4GB would be a good start, and would give us data on whether that was enough.

As for when can it be done, are you asking because this is something which requires a server reboot? If so, do it at the "most convenient time to disrupt the fewest people" - I'm sure IT knows when that is better than I do. When I've done reboots before, I've done them in the morning UK time, so the North Americans are all asleep.

In the past, the issue relating to handing over the app has been the ability to continue to make updates and do testing. If we converted the current api-dev machine to be a staging server, and spun up another one configured identically to be the new production server, a) you could manage that one how you liked, and b) I could continue to do as-near-to-production-as-possible testing with the old one.

Does that sound like a plan?

I am currently working on a number of high priority timestamp and timezone bugs (e.g. bug 610473 and bug 592163). When I've fixed those, I'll be releasing 0.8. That might be a good time to hand over.

Gerv

Comment 3

[matthew zeier [:mrz]]

Yes, adding memory requires a reboot.

I was asking for guidance because you know your app better than I do.

We talked about having you update your code in some repo and my team pulling from that repo into prod.  You were resistant to that at the time.

If I roll this into normal downtime, it won't happen until next Tuesday.

Comment 4

[Bob Clary [:bc] (inactive)]

fwiw, I've disabled the use of BzApi by my workers in phx, sjc and locally until I can rethink the implementation. That should eliminate a significant amount of load for the near term. gerv: how is the machine holding up today?

Comment 5

[Gervase Markham [:gerv]]

bc: no-one skilled has yet taken me up on my wish for someone to read the Munin logs and tell me. But it seems to be alright, although I'm investigating one bug.

Gerv

Comment 6

[matthew zeier [:mrz]]

Appeared to be low/no usage right now - bumped up to 6GB RAM.  You should make sure your kernel can use > 4GB RAM.

Comment 7

[matthew zeier [:mrz]]

Your current kernel only sees 4GB.

Comment 8

[matthew zeier [:mrz]]

Take a look @ https://help.ubuntu.com/community/EnablingPAE#Earlier%20versions%20%28Ubuntu%209.04%20and%20%20earlier%29 .

Comment 9

[Gervase Markham [:gerv]]

4GB seems to be enough for now. We aren't hitting the new memory ceiling. Thanks :-)

Gerv