363292 - win32 tinderbox hangs with connection::get_buffer() unexpected error from select: 10093

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Phil Ringnalda (:philor)]

Around 19:00 2006-12-08 both Firefox Windows tinderboxes stopped building (or at least stopped showing up in showbuilds.cgi). Nobody seemed to be around in #build last night or this morning, and nobody in #developers seemed to have an interest in sheriffing, though 12 hours without Windows builds seems to me like a tree-closer; ccing a few of the most likely to decide about that instead.

Comment 1

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

It may just be one tinderbox, since the second one just does perf tests on the other's builds (and if there aren't any builds, it probably won't do anything).

Comment 2

[:Gijs (out until Jan 5th; he/him)]

It's been long enough for them to completely drop off the page by now.

Comment 3

[Robert Sayre]

shouldn't the tree be closed at this point?

Comment 4

[Robert Helmer [:rhelmer]]

I'll take a look.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> It may just be one tinderbox, since the second one just does perf tests on the
> other's builds (and if there aren't any builds, it probably won't do anything).

That is correct, test-only machines only test when the tinderbox status is "success" and a new file has been published to the hourly dir. 

fx-win32-tbox has been restarted and is reporting now, closing the bug.

Comment 6

[Peter van der Woude [:Peter6]]

Unfortunately it stopped again

Comment 7

[Phil Ringnalda (:philor)]

And, curiously, at the moment sea-win32-tbox has been pretending to build SeaMonkey for seven hours, sb-win32-tbox last built Calendar 12 hours ago, moz180-win32-tbox and tb180-win32-tbox both stopped more than 12 hours ago, and fxexp-win32-tbox stopped trying to build XULRunner more than 24 hours ago. Unless I'm forgetting something, Thunderbird trunk and the 1.8 branch are the only things  with working Windows tinderboxes.

Comment 8

[Robert Helmer [:rhelmer]]

They are all exhibiting the same problem as earlier, looping:

connection::get_buffer() unexpected error from select: 10093
connection::get_buffer() unexpected error: 10093

I'll go through and restart them, I'm expecting it to happen again at this point since it seems to be happening on all of our trunk ref VMs (and happened again after a restart of fx/fxexp by me this afternoon) so we should start looking for the root cause.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #8)
> They are all exhibiting the same problem as earlier, looping:
> 
> connection::get_buffer() unexpected error from select: 10093
> connection::get_buffer() unexpected error: 10093

Here is a breakdown:

* fx-win32-tbox was looping the errors above
* sea-win32-tbox, moz180-win32-tbox, tb180-win32-tbox do not have extra logging on; apparently stopped between builds, reason unknown (possibly same as above)
* sb-win32-tbox seems fine

Restarted tinderbox on all.

Comment 10

[timeless]

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1165751700.495.gz

not very happy

Comment 11

[Peter van der Woude [:Peter6]]

there we go again....
Gone since 5 hours

Comment 12

[J. Paul Reed [:preed]]

(In reply to comment #8)
> They are all exhibiting the same problem as earlier, looping:
> 
> connection::get_buffer() unexpected error from select: 10093
> connection::get_buffer() unexpected error: 10093
> 
> I'll go through and restart them, I'm expecting it to happen again at this
> point since it seems to be happening on all of our trunk ref VMs (and happened
> again after a restart of fx/fxexp by me this afternoon) so we should start
> looking for the root cause.

Did this happen on the testing tinderboxen, or on the build tinderboxen?

I ask because if it's the former, then those aren't really ref VMs, correct? If they are, then I'll help investigate the problem. If they aren't, then we should try to test this condition with the new testing ref VMs which, to my knowledge, don't yet exist.

Comment 13

[(not currently active) Ted Mielczarek]

(In reply to comment #8)
> They are all exhibiting the same problem as earlier, looping:
> 
> connection::get_buffer() unexpected error from select: 10093
> connection::get_buffer() unexpected error: 10093
> 
> I'll go through and restart them, I'm expecting it to happen again at this
> point since it seems to be happening on all of our trunk ref VMs (and happened
> again after a restart of fx/fxexp by me this afternoon) so we should start
> looking for the root cause.

Just for the record, this error appears to be coming from blat:
http://www.google.com/codesearch?hl=en&q=+connection::get_buffer+show:Udf_fs4xs4M:NqTyq18_Bfs:abj9dJt_89g&sa=N&cd=4&ct=rc&cs_p=http://www.enterprisesecuritysolutions.net/files/checksyslog.zip&cs_f=checksyslog/blat194/GENSOCK/gensock.cpp#a0

And the select error is WSAENOTINITIALIZED:
http://support.ipswitch.com/kb/WSK-19980714-EM37.htm

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to comment #12)
> (In reply to comment #8)
> > They are all exhibiting the same problem as earlier, looping:
> > 
> > connection::get_buffer() unexpected error from select: 10093
> > connection::get_buffer() unexpected error: 10093
> > 
> > I'll go through and restart them, I'm expecting it to happen again at this
> > point since it seems to be happening on all of our trunk ref VMs (and happened
> > again after a restart of fx/fxexp by me this afternoon) so we should start
> > looking for the root cause.
> 
> Did this happen on the testing tinderboxen, or on the build tinderboxen?
> 
> I ask because if it's the former, then those aren't really ref VMs, correct? If
> they are, then I'll help investigate the problem. If they aren't, then we
> should try to test this condition with the new testing ref VMs which, to my
> knowledge, don't yet exist.

The build machines; see comment 9.

Comment 15

[Phil Ringnalda (:philor)]

Dropping the severity, since we've now made it through two open-tree weekends without any Windows perf data, and it doesn't seem to have blocked anyone's development and/or testing work.

Comment 16

[Boris Zbarsky [:bzbarsky]]

Uh...  No.  This IS a blocker, and the tree should NOT be open until this bug is resolved.  If it's open now, it needs to be closed.

Just because people suck and check in when we don't have perf metrics doesn't mean doing that is OK; if they can't figure out that a box is missing (because that's hard to realize with tinderbox) we have to close the tree to make it clear that checking in is not allowed.

That said, the machines are up, no?  Why is this bug still open?

Comment 17

[Robert Helmer [:rhelmer]]

We have better monitoring in place now, so [e-mail] and #build are notified as builds fall off the tinderbox page. 

Looks like the root cause of this particular may be blat, I am going to change the summary of this bug to reflect that. Please open a new bug if you see any more tinderbox problems, and we can either point back to this one if it turns out to be the cause or investigate new issues that come up.

Comment 18

[Phil Ringnalda (:philor)]

Don't suppose anyone happens to remember exactly when it was that someone got pissed off about this, and updated blat, and on which machine(s)? I seem to be filing on tb-win32-tbox a fair bit, but haven't filed on fx-win32-tbox in quite a while (which could just be more aggressive monitoring getting to its hangs before I do).

Comment 19

[Chris Cooper [:coop] (he/him)]

Has this recurred recently?

I haven't had to unstick a machine from this condition myself in ~6 months, but server-ops may be getting paged before I even notice it. 

Moving to MozillaBuild has helped us a bit here, but we still have lots of Windows tinderboxes running cygwin+blat. I'd like to close this, but am reluctant to do so if the issue is still lurking.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Seems likely this was fixed by reference platform upgrades.