495407 - Try server talos should dump Tp stats when Tp times out

Status: RESOLVED WORKSFORME

(Testing :: Talos, enhancement, P5)

Description

[Henri Sivonen (:hsivonen)]

Steps to reproduce:
 1) Make try server build a recent rev from http://hg.mozilla.org/users/mrbkap_mozilla.com/html5parsing/

Actual results:
Mac and Linux Talos performs several Tp cycles but time out after about 2 hours and 8 minutes before completing Tp. No per-page times are printed out to be able to diagnose if there's a single insanely slow page in the set.

Expected results:
Expected per-page stats for the completed cycles to be dumped into the log. 

Additional info:
Local testing shows that I should expect the HTML5 repo to beat trunk by a small amount on Tp instead of being so slow that it times out. A previous green Tp run showed the the HTML5 repo somewhat slower than trunk but still of the same order of magnitude. (If I ask try server to rebuild that rev again, the timeout thing happens.)

(Marking as blocker, because this prevents me from doing evidence-based perf work on the HTML5 repo.)

Comment 1

[alice nodelman [:alice] [:anode]]

I believe that this will be a one line change, I'll investigate.

Comment 2

[alice nodelman [:alice] [:anode]]

Here's a tp timeout shows the per-page info after Tp times out:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1244829510.6.1244837927.19308.gz&fulltext=1

Do you have an example of the failure that you are seeing?

Comment 3

[Henri Sivonen (:hsivonen)]

Attachment: Make the app itself dump times

That's the log that shows the failure. It should be reproducible by building rev d2822f44e1c5 from http://hg.mozilla.org/users/mrbkap_mozilla.com/html5parsing/ with the attached patch.

However, it's now understood that the cycles get slower and slower on talos, which suggests that under talos memory conditions (but not under the memory conditions of my dev machine), a leak could be making things significantly slower.

Comment 4

[Henri Sivonen (:hsivonen)]

Downgrading the severity now that there's an actionable hypothesis by other means.

Comment 6

[alice nodelman [:alice] [:anode]]

Putting into future queue, no plans to work on this this quarter.

Comment 7

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

This is a talos functionality enhancement, over to the A-team.

Comment 9

[Jeff Hammel]

Is this bug still applicable?  I think both the timeouts and the dumping have since been fixed for try and elsewhere

Comment 10

[Henri Sivonen (:hsivonen)]

(In reply to Jeff Hammel [:jhammel] from comment #9)
> Is this bug still applicable?  I think both the timeouts and the dumping
> have since been fixed for try and elsewhere

I don’t know. I haven’t had to debug Talos lately.

Comment 11

[Jeff Hammel]

I vote for closing.  Any objections?