978684 - Multiple disconnects on mm-win-81-64-3 with WindowsError: [Error 6] The handle is invalid.

Status: RESOLVED WORKSFORME

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Andrei Eftimie]

We've had around 20 failures over the weekend on mm-win-81-64-3 with:
> 13:37:26 WindowsError: [Error 6] The handle is invalid.
> 13:37:26 
> 13:38:49 IO Completion Port unexpectedly closed

I've attached one of the logs in full.

The machine itself had multiple notifications that said:
> "You have new applications that can open webpages"
One of these is shown whenever we install a new Firefox build. These usually disappear after a few seconds by themselves. I had to manually close all of them. 3-4 of them were visible at a time, and whenever I closed one, a new one would show up. I haven't properly counted all, but there were around 15 if not more in total.

I checked the system logs but didn't find anything suspicious that would correlate with our failures.

For now I've deleted the Jenkins folder and restarted the machine (we had 37 days uptime, which shouldn't be a problem, but this is Windows and as with Windows a reboot usually does wonders).

Maybe we should disable these notifications altogether.
As of now I haven't found a way of doing this.

Comment 1

[Andrei Eftimie]

Attachment: log.txt

Comment 2

[Andreea Matei [:AndreeaMatei]]

I have found this options to disable that notification, if we decide we want to do that:
http://www.eightforums.com/tutorials/26244-new-app-installed-notification-disable-windows-8-a.html

Indeed I don't see why we would want to keep it.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Is this the node that Cosmin ran Metro tests on last week? If so, perhaps that's thew reason for these notifications. Either way, we almost certainly want to disable these notifications on our CI nodes.

Comment 4

[Cosmin Malutan, [:cosmin-malutan]]

No, I ran only twice on mm-win-81-64 staging node.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yeah, I agree that we should disable that setting on the Windows 8 and 8.1 machines. Can someone please file an infrastructure bug for it?

Comment 6

[Andrei Eftimie]

Attachment: mm-win-81-32-1_new-apps.png

We've had another instance over the weekend on mm-win-81-32-1.
All testruns failed with "IO Completion Port unexpectedly closed”.

Not sure if relevant but here's where it started:
> Failed  release-mozilla-beta_functional #7298 // First failure
> Success release-mozilla-beta_addons #7282 // Last good testrun

All testruns after #7298 failed in the same way, usually during the first couple of tests.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Andrei, is that still reproducible? If yes it would be a good chance for me to investigate that.

Comment 8

[Andrei Eftimie]

Not sure what you mean by reproducible.

There was one particular difference between the recent failure and the one initially reported.
The first time I had to manually close each and every Notification.

This time I logged on the machine. In noticed the stacked Notifications. I made the previously attached print screen. I tried clicking on one such Notification but for some reason my VNC client appeared to be frozen. I closed it and restarted the VNC connection. Once I was reconnected there were no Notifications visible.

I didn't had any direct action to clean them up. Rebuild testruns have passed, so probably not reproducible at this time.

I haven't give it much thought since we have bug 979249 that should make this go away.
But that might be a good ideea.

====
Note: if you encounter this issue again, please take the machine offline without any attempt to clean it up, we might be able to investigate the issue.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Eftimie from comment #8)
> Not sure what you mean by reproducible.
[..]
> I didn't had any direct action to clean them up. Rebuild testruns have
> passed, so probably not reproducible at this time.

You are answering yourself here. :) Anyway it might make it hard for us to investigate. But I doubt that this problem is really because of those notifications. The error message doesn't make sense. But lets revisit once it happens again.

Comment 10

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> But I doubt that this problem is really because of those
> notifications. The error message doesn't make sense. But lets revisit once
> it happens again.

This is the 2nd time we've seen this. Both times _all_ testruns on the affected machines failed.
This untill the Notifications were gone. I can't see this as a coincidence.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We haven't seen this again in the last year, so marking as WFM.