419620 - qm-mini-vista04 (branch) numbers gradually rising

Status: RESOLVED DUPLICATE of bug 463020

(Release Engineering :: General, defect, P3)

Description

[Johnathan Nightingale [:johnath]]

It looks like there's some runaway climb on qm-mini-vista04 (our 1.8 vista box).  Ts has gone up by about 20% and Twinopen has doubled over the last three months.

Given how gradual the rise is on a branch box, I'm thinking this is the box's fault, not a real series of regressions.

Comment 1

[alice nodelman [:alice] [:anode]]

I rebooted the machine and cleared out temporary files and such.  The numbers seem to have normalized.

Please re-open if they start to go off the charts again.  We may need to schedule weekly reboots for this box, there must be something in branch that is leaving behind chunder and clogging the machine - otherwise I'd be seeing the same behavior on the other vista boxes.

Comment 2

[Justin Dolske [:Dolske]]

Hmm, I'm a little suspicious of the trunk numbers too. They also show a gradual increase -- though not as extreme as branch -- which seems odd because it's rather constant (and with all the other perf work going on, you'd think these would be trending downward).

Perhaps try flushing out the trunk boxes too, to see if they drop? [And, uhh, is it known why qm-mini-vista01/03 have not reported Twinopen results for 2 days?] 

http://graphs.mozilla.org/#spst=range&spstart=1198092705&spend=1198104712&bpst=cursor&bpstart=1198092705&bpend=1198104712&m1tid=61094&m1bl=0&m1avg=0&m2tid=61269&m2bl=0&m2avg=0&m3tid=61184&m3bl=0&m3avg=0&m4tid=61114&m4bl=0&m4avg=0

Comment 3

[alice nodelman [:alice] [:anode]]

qm-mini-vista01/03 haven't reported because they are sitting idle.  Due to PGO being activated for win builds we are hitting bug 419071.  There's nothing wrong with the machines, builds just aren't being produced fast enough for more talos slaves to be active.

I'll find some time next week to do clean up on the other vista boxes to see if the numbers change at all.

Comment 4

[Mikeal Rogers [:mikeal] (mrogers@)]

I talked to alice about this one.

I'll duplicate the cleaning of temp files and reboot on the trunk machines.

Comment 5

[alice nodelman [:alice] [:anode]]

I did a disk cleanup on qm-mini-vista01-05 (Control Panel -> System Maintenance -> Performance Information and Tools -> Disk Cleanup).  I want to see if we can avoid having to reboot the machines.

Comment 6

[Ryan A. C.]

I think this is vista disk caching gone wrong.
I use a program called PassMark BurnInTest to test all new hardware I put out, and I have noticed that for some reason a vista machine will remain extremely unresponsive long after the tests are stopped. I've narrowed it down to the Ram Torture combined with Hard Disk test, and have noticed the hard disk activity will remain at nearly the same levels as the torture test. Unfortunately I can't suggest any fixes as I generally just reboot the machine before I continue working on it, but I figured you might be running in to the same issue, just much more gradually.

Comment 7

[alice nodelman [:alice] [:anode]]

Took the opportunity during the ongoing netapp outage to reboot these machines.  Obviously not a permanent solution, but should improve things for the short term.  I think that our current options for resolving this outright are:

- as mentioned in comment #6, determine what's up with disk caching and turn it off
- schedule periodic reboots for vista machines

Comment 8

[alice nodelman [:alice] [:anode]]

Rebooting fixes this, so duping to rebooting talos machines at reasonable intervals.