454837 - Investigate sporadic oranges (RLk leaks) on fxdbug-linux-tbox

Status: RESOLVED WORKSFORME

(Core :: Networking, defect)

Description

[Daniel Holbert [:dholbert]]

The Firefox3.0 tinderbox fxdbug-linux-tbox has been having orange bulds due to Rlk leaks every few cycles for quite some time now.

The failures show up on Tinderbox as:
  RLk:92.0B
with logs like:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1221154200.1221154581.13987.gz

These failures are undesirable for a number of reasons...
 - They imply that we've got a sporadic leak, which is be bad.
 - They add noise to the Firefox3.0 tinderbox page
 - They add noise to Firebot spamming #developers about failures

To visualize the noisiness, see this build graph of recent leak stats on that box.  It's highly spiky, jumping back and forth between 0 and 92.
http://build-graphs.mozilla.org/graph/query.cgi?tbox=fxdbug-linux-tbox.build.mozilla.org&testname=refcnt_leaks&autoscale=1&size=&days=1&units=bytes&ltype=lines&points=1&showpoint=2008:09:11:10:36:08,92&avg=1
(not to mention it's also undesirable because it 

I'm filing this bug to investigate this issue.  (I'm not sure if I'm filing it in the right component -- I just used the "Tinderbox Blocker" link from the sheriff wiki page)

Comment 1

[Daniel Holbert [:dholbert]]

(In reply to comment #0)
>  - They imply that we've got a sporadic leak, which is be bad.
I fail at english... s/is be/would be/

Comment 2

[(not currently active) Ted Mielczarek]

I think we probably actually do have a sporadic leak here. I don't know why you'd assume anything else given the evidence. It's leaking a single nsHostResolver, so I bet it's not that hard to figure out where that was introduced.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Obviously not a blocker, if this is the first time someone filed a bug.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: 92 byte leaks

This is from a grep on the raw data from the graph. Looks like it really kicked off around 2008:08:25:00:21:37, but there were intermittent failures before then.

Much as I don't want to disagree with you here Ted, I can't find a checkin that might be responsible in
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=PhoenixTinderbox&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008%2F08%2F18+00%3A16&maxdate=2008%2F08%2F26+00%3A16&cvsroot=%2Fcvsroot

Looks like it was rebooted recently (nearly 7 days ago). I can try a clobber, otherwise something subtle is going on.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Clobbered, lets see what happens. System logs look clean, there were no config changes on the tinderbox (or even anyone logging in!) - all for the period 20th August onwards.

Comment 6

[Daniel Holbert [:dholbert]]

(In reply to comment #3)
> Obviously not a blocker, if this is the first time someone filed a bug.

Sorry -- I used the "File a Tinderbox Maintenance Blocker Ticket" link on the sheriff wiki page, which, yes, I guess sets it up as a blocker. :)

Thanks for investigating.  Hopefully the clobber fixes it...

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The clobber didn't help. I've verified the DNS settings match other boxes, anyone got other ideas or checking to blame ?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Er, checkin. Don't think this is a box problem, over to Core::Networking for code investigation.