861228 - bugs.ecmascript.org database is slow

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Chris Turra [:cturra]]

moving this into a bug... 

-------- Original Message --------
Subject: Re: bugs.ecmascript.org database is slow today
Date: Fri, 12 Apr 2013 07:08:25 -0700
From: matthew zeier <[e-mail]>
To: Brendan Eich <[e-mail]>
CC: Dave Herman <[e-mail]>, "[e-mail] Operations" <[e-mail]>

Probably - Let me take this off your hands and have the WebOps team handle it with Andre.

(this host doesn't fit into the standard, factory IT config so it'll take a bit for someone to investigate)


On Apr 12, 2013, at 7:07 AM, Brendan Eich <[e-mail]> wrote:

> Not good -- ticket time?
> 
> /be
> 
> -------- Original Message --------
> Subject: 	Re: bugs.ecmascript.org database is slow today
> Date: 	Fri, 12 Apr 2013 11:55:59 +0200
> From: 	André Bargull <[e-mail]>
> Reply-To: 	[e-mail]
> To: 	Brendan Eich <[e-mail]>
> CC: 	es-discuss <[e-mail]>
> 
> 
> 
> It improved a bit, I've just added a new bug and the loading time was
> only one minute instead of two minutes.
> 
> - André
> 
> On 4/12/2013 3:21 AM, Brendan Eich wrote:
>> I heard some db "cleaning" was just done -- any improvements?
>> 
>> /be
>> 
>> André Bargull wrote:
>>> A bit off topic:
>>> 
>>> bugs.ecmascript.org is extremely slow today, when I file a new issue,
>>> the browser loads for about two minutes. And I even got multiple
>>> "mid-air collision detection" errors for my own (!) changes. Maybe
>>> someone can take a look and figure out what's going on there. (The
>>> search functionality is not affected.)
>>> 
>>> Thanks,
>>> André

Comment 1

[Chris Turra [:cturra]]

providing an update here - we have reviewed the virtual machine and resource usage doesn't appear to be constrained. i have also chatted with one of our dba's who will work with me on monday to review the mysql setup on this host and do some tuning.

Comment 2

[Sheeri Cabral [:sheeri]]

The db in total has 1441 bugs and is 42M, so it's not a large db, then again this machine also is not very powerful.

Things that will make the DB faster:
- more RAM - the machine has 512 Mb, innodb buffer pool is using 24M
   - though this might not help too much, the innodb buffer pool is using 536 pages, and there are 995 pages free right now.
- Upgrade MySQL (right now the db is at MySQL 5.0.77, which is from 2009, and the entire 5.0 tree is EOL)
- Go innodb_file_per_table (requires a small amount of downtime as we do a mysqldump and restore). This will improve I/O, right now almost all the data and indexes are in one file.

In the error logs I see this:
130411 17:10:40 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295

This is an old bug - http://bugs.mysql.com/bug.php?id=35346 - This also might be slowing things down, we should set it explicitly.

There's no binary or slow logging going on, so you're saving lots of I/O there.

Comment 3

[Chris Turra [:cturra]]

:gcox is going to be adding some more RAM to the VM this evening. once that's complete, :sheeri and i will look at upgrading the MySQL server on this host.

Comment 4

[Greg Cox [:gcox]]

bumped RAM to 1G per email request.

Comment 5

[Chris Turra [:cturra]]

thnx :gcox. 

now that we have increased the RAM on this virtual machine, :sheeri and i are going to perform some upgrades to MySQL and apache tomorrow (06/18) evening at 5:00pm pacific. this will require a little bit of downtime to complete. 

we'll be taking a snapshot of the vm, so we can roll back if these upgrades fail for any reason.

Comment 6

[Sheeri Cabral [:sheeri]]

I have upgraded to MySQL 5.5.

Comment 7

[Chris Turra [:cturra]]

in addition to the mysql updates noted by :sheeri in comment 6, i went ahead and updated apache and a number of other system packages (including the kernel). 

before we did these upgrades i was able to reproduce the extreme slowness while opening a new account. after these upgrades, opening a new bugzilla account was *quick* once again.

looks like all our package upgrades solved the slowness :)

Comment 8

[Chris Turra [:cturra]]

seems slow again this morning.

Comment 9

[Chris Turra [:cturra]]

this looks to have been related to our name server (resolv.conf) configuration. last night when i saw this fast, i had enabled google's public dns servers (8.8.8.8) for the yum updates. when i put things back i probably didn't complete my testing thoroughly enough. 

this morning i poked around at the resolv.conf and it turns out a couple of the name servers and it looks like a couple appear to be timing out (thus causing this slowness). i have removed them from the list and things are snappy again!

i am going to follow up with netops to sort out what name servers *should* be used, but at this point, everything should be back to normal with this site :D