551420 - Error logs filling up disk on generic cluster webheads

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect)

Description

[Shyam Mani [:fox2mike]]

Hey folks,

Over the last couple of days, I've noticed that we've been getting nagios pages for disk space on the generic cluster like this :

02:08:43 <@nagios> [55] pm-app-generic06:root partition is WARNING: DISK WARNING - free space: / 6563 MB (9% inode=86%):

Upon further investigation, I found out that the culprit is actually the amount of error logs being dumped by SUMO..and would like you guys to take a look and see how we can fix this ASAP.

Sample log :

[Wed Mar 10 02:45:04 2010] [error] [client 10.2.81.104] File does not exist: /data/www/support.mozilla.com/webroot/styles/ie.css, referer: http://support.mozilla.com/search.php?where=all&locale=en-US&qs=s&q=&sa=Search
[Wed Mar 10 02:45:04 2010] [error] [client 10.2.81.100] File does not exist: /data/www/support.mozilla.com/webroot/styles/ie.css, referer: http://support.mozilla.com/search.php?q=downloads&where=all&locale=en&qs=h
[Wed Mar 10 02:45:04 2010] [error] [client 10.2.81.101] File does not exist: /data/www/support.mozilla.com/webroot/styles/mozms2/btn_bg.jpg, referer: http://support.mozilla.com/minify.php/css_styles_mozcommon;css_styles_mozms2
[Wed Mar 10 02:45:04 2010] [error] [client 10.2.81.100] File does not exist: /data/www/support.mozilla.com/webroot/styles/mozms2/btn_bg.jpg, referer: http://support.mozilla.com/minify.php/css_styles_mozcommon;css_styles_mozms2
[Wed Mar 10 02:45:04 2010] [error] [client 10.2.81.102] File does not exist: /data/www/support.mozilla.com/webroot/styles/mozms2/btn_bg.jpg, referer: http://support.mozilla.com/minify.php/css_styles_mozcommon;css_styles_mozms2
[Wed Mar 10 02:45:05 2010] [error] [client 10.2.81.104] File does not exist: /data/www/support.mozilla.com/webroot/styles/mozms2/btn_bg.jpg, referer: http://support.mozilla.com/minify.php/css_styles_mozcommon;css_styles_mozms2
[Wed Mar 10 02:45:05 2010] [error] [client 10.2.81.101] File does not exist: /data/www/support.mozilla.com/webroot/styles/mozms2/btn_bg.jpg, referer: http://support.mozilla.com/minify.php/css_styles_mozcommon;css_styles_mozms2

Comment 1

[Laura Thomson :laura]

Most of those seem to be covered by bug 549978, I'll look into the ie.css one.

Comment 2

[Laura Thomson :laura]

Attachment: itsy bitsy patch #2

Comment 3

[Paul Craciunoiu [:paulc]]

Comment on attachment 431683 [details] [diff] [review]
itsy bitsy patch #2

It looks like each theme has its own ie(7).css stylesheet. This patch would prevent loading it.

See:
http://viewvc.svn.mozilla.org/vc/projects/sumo/trunk/webroot/styles/mozcommon/ie.css?view=markup

vs

http://viewvc.svn.mozilla.org/vc/projects/sumo/trunk/webroot/styles/mozms2/ie.css?view=markup

(they're different)

It looks like the errors show up when {$theme_dir} is empty. Would the logs show what's the URL that requests this stylesheet? Perhaps we could figure out why it's empty that way. My guess is that it happens when the directory does not exist.

Comment 4

[Laura Thomson :laura]

Whoops, my bad.  

Yeah, it's the search URL (see above) so I'd better check the theme for search.

Comment 5

[Laura Thomson :laura]

Summary: search uses mozsadvanced, which doesn't have its own themedir.

Solution: symlink a themedir to whichever of the other themes (mozkb? probably) has a stylesheet that works with ie6/ie7.

Passing off to paulc because he has IE6 for testing and I don't (woot)

Comment 6

[Paul Craciunoiu [:paulc]]

Attachment: assign theme_dir in search.php

This should do it. Tested in IE6/7, layout looks the same AFAICT.

Also, I did a diff on the output and this is the only change:

<     <!--[if lte IE 6]><link rel="StyleSheet" media="all" href="/styles/mozkb/ie.css" type="text/css" /> <![endif]-->
<     <!--[if IE 7]><link rel="StyleSheet" media="all" href="/styles/mozkb/ie7.css" type="text/css" /> <![endif]-->
---
>     <!--[if lte IE 6]><link rel="StyleSheet" media="all" href="/styles//ie.css" type="text/css" /> <![endif]-->
>     <!--[if IE 7]><link rel="StyleSheet" media="all" href="/styles//ie7.css" type="text/css" /> <![endif]-->

Comment 7

[Paul Craciunoiu [:paulc]]

r64026/r64027 (trunk+fennec)

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on SUMO trunk using:

* Firefox 3.6
* IE 6, 7, 8
* Opera 10.10
* Safari 4.0.4
* Google Chrome 5

After I'm through with this meeting, I'll test on the Fennec staging site and verify.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; we'll retest on Fennec's staging branch in 1.5.3.