bugzilla.mozilla.org will be intermittently unavailable on Saturday, March 24th, from 16:00 until 20:00 UTC.

bugs.ecmascript.org database is slow



Infrastructure & Operations
WebOps: Other
5 years ago
5 years ago


(Reporter: cturra, Assigned: cturra)





5 years ago
moving this into a bug... 

-------- Original Message --------
Subject: Re: bugs.ecmascript.org database is slow today
Date: Fri, 12 Apr 2013 07:08:25 -0700
From: matthew zeier <mrz@mozilla.com>
To: Brendan Eich <brendan@mozilla.com>
CC: Dave Herman <dherman@mozilla.com>, "infra-webops@mozilla.com Operations" <infra-webops@mozilla.com>

Probably - Let me take this off your hands and have the WebOps team handle it with Andre.

(this host doesn't fit into the standard, factory IT config so it'll take a bit for someone to investigate)

On Apr 12, 2013, at 7:07 AM, Brendan Eich <brendan@mozilla.com> wrote:

> Not good -- ticket time?
> /be
> -------- Original Message --------
> Subject: 	Re: bugs.ecmascript.org database is slow today
> Date: 	Fri, 12 Apr 2013 11:55:59 +0200
> From: 	André Bargull <andre.bargull@udo.edu>
> Reply-To: 	andre.bargull@udo.edu
> To: 	Brendan Eich <brendan@mozilla.com>
> CC: 	es-discuss <es-discuss@mozilla.org>
> It improved a bit, I've just added a new bug and the loading time was
> only one minute instead of two minutes.
> - André
> On 4/12/2013 3:21 AM, Brendan Eich wrote:
>> I heard some db "cleaning" was just done -- any improvements?
>> /be
>> André Bargull wrote:
>>> A bit off topic:
>>> bugs.ecmascript.org is extremely slow today, when I file a new issue,
>>> the browser loads for about two minutes. And I even got multiple
>>> "mid-air collision detection" errors for my own (!) changes. Maybe
>>> someone can take a look and figure out what's going on there. (The
>>> search functionality is not affected.)
>>> Thanks,
>>> André

Comment 1

5 years ago
providing an update here - we have reviewed the virtual machine and resource usage doesn't appear to be constrained. i have also chatted with one of our dba's who will work with me on monday to review the mysql setup on this host and do some tuning.
The db in total has 1441 bugs and is 42M, so it's not a large db, then again this machine also is not very powerful.

Things that will make the DB faster:
- more RAM - the machine has 512 Mb, innodb buffer pool is using 24M
   - though this might not help too much, the innodb buffer pool is using 536 pages, and there are 995 pages free right now.
- Upgrade MySQL (right now the db is at MySQL 5.0.77, which is from 2009, and the entire 5.0 tree is EOL)
- Go innodb_file_per_table (requires a small amount of downtime as we do a mysqldump and restore). This will improve I/O, right now almost all the data and indexes are in one file.

In the error logs I see this:
130411 17:10:40 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295

This is an old bug - http://bugs.mysql.com/bug.php?id=35346 - This also might be slowing things down, we should set it explicitly.

There's no binary or slow logging going on, so you're saving lots of I/O there.

Comment 3

5 years ago
:gcox is going to be adding some more RAM to the VM this evening. once that's complete, :sheeri and i will look at upgrading the MySQL server on this host.

Comment 4

5 years ago
bumped RAM to 1G per email request.

Comment 5

5 years ago
thnx :gcox. 

now that we have increased the RAM on this virtual machine, :sheeri and i are going to perform some upgrades to MySQL and apache tomorrow (06/18) evening at 5:00pm pacific. this will require a little bit of downtime to complete. 

we'll be taking a snapshot of the vm, so we can roll back if these upgrades fail for any reason.
I have upgraded to MySQL 5.5.

Comment 7

5 years ago
in addition to the mysql updates noted by :sheeri in comment 6, i went ahead and updated apache and a number of other system packages (including the kernel). 

before we did these upgrades i was able to reproduce the extreme slowness while opening a new account. after these upgrades, opening a new bugzilla account was *quick* once again.

looks like all our package upgrades solved the slowness :)
Last Resolved: 5 years ago
Resolution: --- → FIXED

Comment 8

5 years ago
seems slow again this morning.
Resolution: FIXED → ---

Comment 9

5 years ago
this looks to have been related to our name server (resolv.conf) configuration. last night when i saw this fast, i had enabled google's public dns servers ( for the yum updates. when i put things back i probably didn't complete my testing thoroughly enough. 

this morning i poked around at the resolv.conf and it turns out a couple of the name servers and it looks like a couple appear to be timing out (thus causing this slowness). i have removed them from the list and things are snappy again!

i am going to follow up with netops to sort out what name servers *should* be used, but at this point, everything should be back to normal with this site :D
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.