388054 - Investigate spikes in Tp/Tp2/etc. on fxnewref-linux-tbox perf test tinderbox

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Adam Guthrie]

There are pretty weird spikes in the times of almost all perf. tests. For example, Tp:

http://build-graphs.mozilla.org/graph/query.cgi?testname=pageload&units=ms&tbox=bl-bldlnx03_fxnewref-linux-tbox-HEAD&autoscale=1&days=7&avg=1&showpoint=2007:07:13:09:41:51,265

Is some cron job running? These spikes make it very hard to see regressions by looking at graphs, as they distort the graph so much.

Comment 1

[Robert Helmer [:rhelmer]]

I don't see any cron jobs set up (not this frequently anyway).

One thing I noticed is that yum-updatesd was still running (probably because the service was typo'd as yum-updated in the doc - http://wiki.mozilla.org/ReferencePlatforms/Test/Linux_FC6_(Tinderbox)#Some_Sys-prepping).

I'll keep an eye on this machine and modify http://wiki.mozilla.org/ReferencePlatforms/Test/Linux_FC6_(Tinderbox) as well as this bug when I find out what's going on.

Comment 2

[Robert Helmer [:rhelmer]]

Disabled the following daily/weekly cronjobs (by moving them to cron.*.disabled/), leaving only 0anacron:

cron.daily/0anacron
cron.daily/logrotate
cron.daily.disabled
cron.daily.disabled/prelink
cron.daily.disabled/makewhatis.cron
cron.daily.disabled/tmpwatch
cron.daily.disabled/0logwatch
cron.daily.disabled/cups
cron.daily.disabled/rpm
cron.daily.disabled/mlocate.cron
cron.daily.disabled/beagle-crawl-system
cron.monthly/0anacron
cron.weekly/0anacron
cron.weekly.disabled
cron.weekly.disabled/makewhatis.cron

Comment 3

[Robert Helmer [:rhelmer]]

Note that I left logrotate on as well (so system logs don't pile up); it should be easier to see now what is causing the problem since that's the only task that's likely to have any work to do. I'll disable it if necessary.

Comment 4

[J. Paul Reed [:preed]]

(In reply to comment #1)
> I don't see any cron jobs set up (not this frequently anyway).
> 
> One thing I noticed is that yum-updatesd was still running (probably because
> the service was typo'd as yum-updated in the doc -
> http://wiki.mozilla.org/ReferencePlatforms/Test/Linux_FC6_(Tinderbox)#Some_Sys-prepping).

This was typo'd (actually, they renamed it in CentOS 5) in the doc, but NOT in the sysprep script I used for prepping the VM. This was my bad, though, because you used the wiki (which was exactly correct).

We should add that to the docs to use the CD instead of what's in the wiki (they should match, but we can check the CD in, so that's easier).

Can you include the commands your ran (verbatim) so I can add them in the sysprep script?

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> (In reply to comment #1)
> > I don't see any cron jobs set up (not this frequently anyway).
> > 
> > One thing I noticed is that yum-updatesd was still running (probably because
> > the service was typo'd as yum-updated in the doc -
> > http://wiki.mozilla.org/ReferencePlatforms/Test/Linux_FC6_(Tinderbox)#Some_Sys-prepping).
> 
> This was typo'd (actually, they renamed it in CentOS 5) in the doc, but NOT in
> the sysprep script I used for prepping the VM. This was my bad, though, because
> you used the wiki (which was exactly correct).
> 
> We should add that to the docs to use the CD instead of what's in the wiki
> (they should match, but we can check the CD in, so that's easier).
> 
> Can you include the commands your ran (verbatim) so I can add them in the
> sysprep script?

Sure, for this it was just:
chkconfig yum-updatesd off
service yum-updatesd off

I added it to the "for" loop that turns off services in the wiki.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> (In reply to comment #4)
> > Can you include the commands your ran (verbatim) so I can add them in the
> > sysprep script?
> 
> Sure, for this it was just:
> chkconfig yum-updatesd off
> service yum-updatesd off
> 
> I added it to the "for" loop that turns off services in the wiki.

I've also added the commands used for moving the unwanted jobs to ".disabled" directories (not a direct copy/paste, please sanity check before using :) ).

Comment 7

[Adam Guthrie]

All graphs look VERY stable with no spikes. Looks like this fixed it. (I think I might file another bug for the OS X perf. box, which seems to be experiencing similar issues.) ->FIXED