622353 - Busted DNS on moz2-darwin9-slave19?

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

The only one of https://build.mozilla.org/buildapi/recent/moz2-darwin9-slave19 that I can see was http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1293908089.1293908094.23049.gz where it suffered a "Temporary failure in name resolution" on the first hg clone step, but the 0 to 2 second runtime on all the fuzzer runs makes me think it's probably failing DNS there, too.

Comment 1

[Mike Taylor [:bear]]

turned off buildslave for now - it was just spinning out bad fuzzer jobs

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

zandr: can you verify DNS records for this machine?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It's not a question of the DNS records for this machine - the machine itself cannot resolve names.  I suspect it's a mach context problem -- we had a similar thing happen in bug 607780.  I'll have a look with BootstrapDump (c.f. bug 622556), but I suspect that the result will be the same.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Hmm, the contexts look fine:

0x0
  launchd (root, 1->0)
  update (root, 25->1)
  launchd (cltbld, 82->1)
  launchd (nagios, 217->1)
0x50b
  kextd (root, 10->1)
  notifyd (root, 11->1)
  DirectoryService (root, 12->1)
  syslogd (root, 13->1)
  configd (root, 14->1)
  distnoted (daemon, 15->1)
  mDNSResponder (_mdnsresponder, 16->1)
  securityd (root, 20->1)
  ntpd (root, 24->1)
  RFBRegisterMDNS (root, 29->1)
  mds (root, 30->1)
  KernelEventAgent (root, 32->1)
  hidd (root, 34->1)
  fseventsd (root, 35->1)
  dynamic_pager (root, 37->1)
  diskarbitrationd (root, 40->1)
  autofsd (root, 44->1)
  socketfilterfw (root, 45->1)
  coreservicesd (root, 54->1)
  WindowServer (_windowserver/root, 58->1)
  ocspd (root, 60->1)
  coreaudiod (root, 73->1)
  VNCPrivilegeProx (root, 75->1)
  launchproxy (nagios, 214->1)
  launchproxy (root, 231->1)
  0x20b
    mdworker (cltbld, 99->1)
    mdworker (nobody, 122->1)
  0x30b
    0x120b
      sshd (root, 232->231)
      sshd (cltbld, 234->232)
      bash (cltbld, 236->234)
      BootstrapDump (root, 252->236)
    0x413
      loginwindow (cltbld/root, 31->1)
      AirPort Base Sta (cltbld, 87->82)
      ARDAgent (cltbld/root, 90->82)
      Spotlight (cltbld, 92->82)
      UserEventAgent (cltbld, 93->82)
      pboard (cltbld, 94->82)
      ATSServer (cltbld, 96->82)
      AppleVNCServer (cltbld/root, 97->90)
      Dock (cltbld, 98->82)
      SystemUIServer (cltbld, 100->82)
      Finder (cltbld, 101->82)
      quicklookd (cltbld, 114->82)
      python (cltbld, 213->1)


The DNS settings (via scutil --dns) look correct and are identical to those on moz2-darwin9-slave20.

I'll hook this up to staging and see what I can figure out.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

It seems to be running fine in staging.  I forgot that fuzzer builds do not restart, and the failed reftest build also did not restart because of the failed checkout:

python: can't open file 'tools/buildfarm/maintenance/count_and_reboot.py': [Errno 2] No such file or directory

So I think this was a transient thing - perhaps Ben started this slave up manually, forgetting that you can't do that on OSX?

I've moved it back into production and will keep an eye on it for a bit, but I don't expect any more trouble.