520512 - Need to examine a Leopard Talos machine

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Markus Stange [:mstange]]

I'd like to track down the cause of bug 519893. Basically, that bug is causing us to spend 20% of our startup time in dark matter, at least on our Talos machines.

Since I've repeatedly failed to reproduce the effect on my own machine, it would be great if we could use one of our Leopard Talos machines that show the effect for investigation.

Basically, what I'd like you to do is:
 1. Download this build: http://tests.themasta.com/firefox-3.7a1pre.en-US.mac-shark-tsweirdness.dmg
 2. Start Shark on the Talos machine.
 3. Set the sample interval to the smallest possible value.
 4. Enable Sampling -> Programmatic (Remote).
 5. Run the build through the normal ts perf test, emulating normal conditions
    as closely as possible.
 6. Send me the resulting Shark profile(s).

This is a Shark-enabled build that causes Shark to profile exactly the timeframe that I'm interested in.

Is this feasible? If so, when would you have time to do it?

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> I'd like to track down the cause of bug 519893. Basically, that bug is causing
> us to spend 20% of our startup time in dark matter, at least on our Talos
> machines.
> 
> Since I've repeatedly failed to reproduce the effect on my own machine, it
> would be great if we could use one of our Leopard Talos machines that show the
> effect for investigation.
> 
Are you able to reproduce this problem on TryServer talos machines? If so, let us know when you do, and what machine it is; we'll pull it out of circulation, and give you access.

Comment 2

[Markus Stange [:mstange]]

(In reply to comment #1)
> Are you able to reproduce this problem on TryServer talos machines?

I am; http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1254465836.1254467236.30214.gz&fulltext=1 is one example. (There are boxes with numbers in that log; whenever the number is high, there's something wrong.)

The problem seems to be consistently reproducible on all TryServer talos machines.

Comment 3

[alice nodelman [:alice] [:anode]]

I've taken qm-pleopard-try06 out of the pool.  Please ping me on irc for the login credentials.

Once you hand it back it will be re-imaged and put back in the pool.

Comment 4

[Markus Stange [:mstange]]

I'm done with the machine now. Thanks a lot, having access to it was extremely helpful in tracking down the cause of the phenomenon I've been hunting.

I'll hand the machine to David Anderson for bug 521072.

(It turns out that the phenomenon occurs because the Talos machines don't have screens connected to them. If I disconnect the screen of the Mac Mini we have here in the NZ office, I can reproduce it.)

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> I'm done with the machine now. Thanks a lot, having access to it was extremely
> helpful in tracking down the cause of the phenomenon I've been hunting.
Cool.

> I'll hand the machine to David Anderson for bug 521072.
k, thanks.

> (It turns out that the phenomenon occurs because the Talos machines don't have
> screens connected to them. If I disconnect the screen of the Mac Mini we have
> here in the NZ office, I can reproduce it.)
Really good point. Next time we have someone unable to reproduce a talos problem, we should ask them to try disconnecting their screen, and see if that helps. Thanks for the info, Markus.

Comment 6

[alice nodelman [:alice] [:anode]]

All done here, request for re-image filed.  (bug 521343)