616520 - SeaMonkey Linux leak test builds hanging in alive_test_2 since December 1

Status: VERIFIED FIXED

(SeaMonkey :: Testing Infrastructure, defect)

Description

[Robert Kaiser]

Since December 1, we're seeing Linux leak test builds on trunk hang in alive_test_2 and needing manual interruption to come back.

The revisions from the last good and first bad builds tells us that no c-c checkin has caused it, the m-c regression frame is http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e1983b9db75d&tochange=01bdda1252f0

Incidentally, I've seen Linux builds on my machines suddenly start up in offline mode in the last days, if there's a connection, then bug 614286 from ehsan sounds very likely to have affected this. Therefore, CCing him for a potential clue on what could be happening.

Comment 1

[Robert Kaiser]

As a note, logs of the builds don't show anything strange, see e.g. http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1291343829.1291388374.29231.gz which I killed after hanging for a significant time in this test.

I've meanwhile clobbered all our builders for this, but the first clobbered run didn't show a difference and had to be manually interrupted again.

Comment 2

[(no longer active)]

Does SeaMonkey ship the this service? "@mozilla.org/network/network-link-service;1"

Comment 3

[Robert Kaiser]

(In reply to comment #2)
> Does SeaMonkey ship the this service?
> "@mozilla.org/network/network-link-service;1"

It seems to work locally to instantiate it and fetch .isLinkUp like the test added in attachment 493880 [details] [diff] [review] does. Also, If I get it right, that's an xpcshell-test and we have green xpcshell-tests on those builds. I checked an saw that this might go through the dbus service, and I see that one being built in our build logs.

Comment 4

[Robert Kaiser]

BTW, I can confirm that my local Firefox builds start in online mode, while my SeaMonkey builds as well as my Thunderbird builds start in offline mode.
What does Firefox do differently than Thunderbird and SeaMonkey here?

Comment 5

[(no longer active)]

Can someone point me to the mozconfig file used to build SM tinderbox builds?  I bet this is my fault, a regression from bug 614958.

Comment 6

[(no longer active)]

So, when I start a locally built SeaMonkey, it starts in online mode without any problems...

Robert, how can I run the failing test locally?

Comment 7

[Robert Kaiser]

(In reply to comment #5)
> Can someone point me to the mozconfig file used to build SM tinderbox builds? 

http://hg.mozilla.org/build/buildbot-configs/file/3125d448b587/seamonkey/linux/comm-central-trunk/debug/mozconfig is the one used by those builds (not that the PGO script is pointed to but we don't build with PGO, actually).

(In reply to comment #6)
> So, when I start a locally built SeaMonkey, it starts in online mode without
> any problems...

Fun, I wonder why mine don't (on two different machines).

> Robert, how can I run the failing test locally?

buildbot says that this is what is being called there:

python leaktest.py -- -CreateProfile default
 in dir /builds/slave/comm-central-trunk-linux-debug/build/objdir/mozilla/_leaktest

I've never actually tried that test itself locally.

Comment 8

[(no longer active)]

Hmm, is this behavior Linux only?  I've been testing on Mac, but I could try Linux too, if needed.

I just ran the leaktest suite locally without any problems...

Comment 9

[Robert Kaiser]

(In reply to comment #8)
> Hmm, is this behavior Linux only?

Yes, this is Linux only, sorry I just noticed that the summary missed that, even though the comments mentioned it. Mac (both 32bit and 64bit) and Windows work in this respect, it's somehow only Linux (only tested 32bit, but I'd suspect it happens for 64bit as well) that breaks.

Comment 10

[Mark Banner (:standard8)]

(In reply to comment #4)
> BTW, I can confirm that my local Firefox builds start in online mode, while my
> SeaMonkey builds as well as my Thunderbird builds start in offline mode.
> What does Firefox do differently than Thunderbird and SeaMonkey here?

Thunderbird, and I think SeaMonkey, have some code for managing the online/offline state depending on user prefs. This is the Thunderbird version:

http://mxr.mozilla.org/comm-central/source/mail/base/content/mail-offline.js#41

offline.autoDetect is off for SeaMonkey, and off for Thunderbird only on Linux.

Maybe there's something about the status that's getting confused because ehsan's changed the default to offline, then it presumably comes online during startup. So could that be sending a "network:*" notification that we're not expecting?

Comment 11

[(no longer active)]

The real issue here is that now IO Service defaults to being offline.  It's OK if the network link service is available (see comment 2), but for the cases which it's not (e.g., bug 617389), we start up in offline mode, which is not good.

I think we should fix this at the Necko level.  This is not really something which should be fixed in Thunderbird/SeaMonkey code.  I have a patch which fixes this.  I'm trying to write a test for it now...

Comment 12

[(no longer active)]

Attachment: Patch (v1) - commited

Comment 13

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 496063 [details] [diff] [review]
Patch (v1) - commited

s/if/whether/ and looks great.

Comment 14

[Robert Kaiser]

(In reply to comment #10)
> Thunderbird, and I think SeaMonkey, have some code for managing the
> online/offline state depending on user prefs.

Wow, that's crufty. Shouldn't we clean this up now that there are generic things to handle online/offline status?

> offline.autoDetect is off for SeaMonkey, and off for Thunderbird only on Linux.

Shouldn't we turn it on on all platforms (or remove it and let ioService just do it on its own discretion) now that even Linux can detect the online/offline state (via DBus)?

Comment 15

[Mark Banner (:standard8)]

(In reply to comment #14)
> (In reply to comment #10)
> > Thunderbird, and I think SeaMonkey, have some code for managing the
> > online/offline state depending on user prefs.
> 
> Wow, that's crufty. Shouldn't we clean this up now that there are generic
> things to handle online/offline status?

AFAIK they extend the generic things for online/offline status. I'm pretty sure that core doesn't cover those preferences that we have.

> > offline.autoDetect is off for SeaMonkey, and off for Thunderbird only on Linux.
> 
> Shouldn't we turn it on on all platforms (or remove it and let ioService just
> do it on its own discretion) now that even Linux can detect the online/offline
> state (via DBus)?

Last I heard DBus wasn't very reliable at detecting online/offline state, hence the reason it was turned off.

Comment 16

[Robert Kaiser]

(In reply to comment #15)
> (In reply to comment #14)
> AFAIK they extend the generic things for online/offline status. I'm pretty sure
> that core doesn't cover those preferences that we have.

There at least seems to be some equivalent of the .autoDetect one, if I saw things correctly in the other bug this one is blocking.

Comment 17

[Ben Bucksch (:BenB)]

Comment on attachment 496063 [details] [diff] [review]
Patch (v1) - commited

Commited as <http://hg.mozilla.org/mozilla-central/rev/e68363d92389>

Comment 18

[(no longer active)]

Thanks for pushing the patch, Ben!

Comment 19

[Robert Kaiser]

verified in terms of the hang on leaktest being gone.