355192 - No Mac trunk Firefox builds since 9/29/2006

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

Something seems to have happened to xserve03's trunk Firefox builds: since the one that started at 18:45 on 9/29/2006, it's been crashing on tests, and not delivering nightlies. Doesn't look like it ought to be checkin-related crashing, since the first failed build only picked up a little js/, which hasn't bothered anyone else building trunk.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I'm guessing this should be more serious than "normal" ;)

Comment 2

[Phil Ringnalda (:philor)]

Though in truth, I have no way of knowing whether or not that final checkin, for bug 354392, actually broke something in the perf tests, but only in Fx/Mac. My understanding of the old days, when we actually had sheriffs, is that we would have long since closed the tree and backed it out, or determined that it was something else and at least starred a build with some sort of comment, but apparently the two strikes of Fx and Mac, and the diffused sheriff responsibility, mean it now has to be very obvious, or maybe a more important platform.

cc:ing those responsible, though, just to be nuisance :)

Oh, and for anyone keeping track: we now have a few minutes shy of 96 hours of checkins for which we have no perf data on Fx/Mac, though the Cairo-Cocoa build shows some interesting variation through that period.

Comment 3

[Brian Crowder]

gavin tested out removing this patch without any positive results, so I think brendan and I are off the hook (at least for this particular checkin).

Comment 4

[Samuel Sidler (old account; do not CC)]

I'm strongly for closing the tree until this is fixed. No nightlies mean lots of checkins and no way to check for regressions during this time.

This is a critical bug, imo.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I recommend somebody with access to the build machine VNC in and make sure nothing weird is going on -- e.g., extra processes running, extra windows around, that the tinderbox was last started from the console, etc.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Tree closed.

Comment 7

[Brian Crowder]

Having the callstack of these crashes (since that's what they appear to be getting reported as) would be handy, can someone add those to the bug?

Comment 8

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Tp:[CRASH] just means that the test didn't complete.  There isn't necessarily a crash.

However, if there's not anything obviously wrong with the build machine, the next step is to watch the test run and see what's wrong.

Comment 9

[Robert Helmer [:rhelmer]]

I started the build from the console so we can actually see what's going on. Based on past builds, it'll take ~2 hours.

Comment 10

[Robert Helmer [:rhelmer]]

Attachment: make axolotl (pageserver) -> axolotl.mozilla.org

For some reason Minefield can't load "http://axolotl" anymore, it does a google search instead.

Loading "http://axolotl.mozilla.org" does work; this patch to the tinder-config.pl should enable the test to work. Requesting review from dbaron as ping, nslookup and host are all able to resolve "axolotl", is this a problem with DNS resolution in app? If so we may not want to apply this patch since AFAICT this *should* work.

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Does 'host axolotl' work from the machine?  Could the DNS configuration have changed?

Comment 12

[Robert Helmer [:rhelmer]]

(In reply to comment #11)
> Does 'host axolotl' work from the machine?  Could the DNS configuration have
> changed?
> 

Yes "host axolotl" works on the machine; it has not changed as far as I know, but it certainly could be a machine config problem.

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to comment #12)
> (In reply to comment #11)
> > Does 'host axolotl' work from the machine?  Could the DNS configuration have
> > changed?
> > 
> 
> Yes "host axolotl" works on the machine; it has not changed as far as I know,
> but it certainly could be a machine config problem.

This is a problem with axolotl with the virtual host setup; it's returning litmus when you ask for Host: axolotl

Applying this patch so we can get this going while the axolotl bug is tracked down.

Comment 14

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Further debugging from rhelmer showed that since axolotl hosted litmus temporarily after the server litmus was on went down, something broke so that when given "Host: axolotl" it serves litmus, but "Host: axolotl.mozilla.org" serves pageload content.

Not sure why you'd see google content, though.

Comment 15

[Robert Helmer [:rhelmer]]

(In reply to comment #14)
> Further debugging from rhelmer showed that since axolotl hosted litmus
> temporarily after the server litmus was on went down, something broke so that
> when given "Host: axolotl" it serves litmus, but "Host: axolotl.mozilla.org"
> serves pageload content.
> 
> Not sure why you'd see google content, though.
> 

Actually looks like the google thing was a cut&paste anomaly; I got some junk at the beginning of the URL and the browser didn't recognize it as a URL.
The litmus result that I saw makes sense now though.

I've applied that patch; I am running the tinderbox in --testonly mode to see if the tests go green now.

Comment 16

[Jesse Ruderman]

> Actually looks like the google thing was a cut&paste anomaly

probably bug 355311

Comment 17

[David G King]

Is this still a Smoketest Blocker?

Comment 18

[:Gavin Sharp [email: [e-mail]]]

The bug as summarized is fixed, but I think it was left open to track down the problem with axolotl (per comment 13). I think it would be less confusing to open a new (less critical) bug for that issue, and close this one.

Comment 19

[Robert Helmer [:rhelmer]]

(In reply to comment #18)
> The bug as summarized is fixed, but I think it was left open to track down the
> problem with axolotl (per comment 13). I think it would be less confusing to
> open a new (less critical) bug for that issue, and close this one.


Good idea.. I filed bug 357849 to track that issue.