Closed Bug 904643 Opened 11 years ago Closed 11 years ago

Jenkins slave node mm-win-8-32-3 keeps Firefox instances running

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

Product:
Mozilla QA Graveyard
Component:
Infrastructure
Platform:
x86
Windows 8
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: whimboo, Assigned: whimboo)

Details

As I have seen while investigating recent failures for broken testruns on the mm-win-8-64-3 node, we keep a zombie Firefox process running. Not sure what is causing this problem. But one of the testruns does not cleanly shutdown Firefox. First I thought about update tests, but seeing the history of jobs of this node, it's probably not the case. The last successful run on this node were the esr17 remote tests. After that any testrun directly aborted with the jsbridge timeout because the zombie process keeps the jsbridge port open. All, I will re-enable this node but whenever we hit this problem again we should investigate the problem immediately.
We had this problem again with exactly that machine yesterday. Processes were stacking up and were never killed. So we had about 6 processes running, which were Nightly and Beta (from ondemand tests). Not sure what's wrong with that specific machine, but we really might want to replace the VM. For now I will disable the node again, and will check that on Monday.
I did all the remaining OS and other software updates. Since connecting two of those nodes to our CI on Friday, no crash has happened so far. So I have connected all of our Windows 8.1 nodes now. Lets hope it will stick this time. I will call this fixed for now. Lets reopen if it is still not fixed.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Ups, wrong bug.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I wrongly filed this bug for the 64bit machine, while it was indeed the 32bit one. I have taken the mm-win-8-32-3 machine offline beginning this week. I tried several things but I have absolutely no idea what's wrong with it. Something in the whole system must have been screwed up. Adrian, would you be so kind and re-create the VM with the name mm-win-8-32-3.qa.scl3.mozilla.com from the template? Given that all the other VMs are working as expected, the template has to be fine, and we hopefully shouldn't see the problem afterward. Thanks!
Flags: needinfo?(afernandez)
Hardware: x86_64 → x86
Summary: Jenkins slave node mm-win-8-64-3 keeps Firefox instances running → Jenkins slave node mm-win-8-32-3 keeps Firefox instances running
mm-win-8-32-3.qa.scl3.mozilla.com has been re-deployed. VM was simply turned on, no NEW Windows updates were applied as to leave the VM as untouched as possible so that you could note and track any issues you may experience.
Flags: needinfo?(afernandez)
Forgot to add, previous VM was turned off and renamed to: mm-win-8-32-3.qa.scl3.mozilla.com_904643 in case you need anything (will be deleted in 7 days).
Thanks Adrian. I checked everything and except that Java wanted to check for updates automatically again, all is fine. I might want to update that in the template the next time we have to convert it. But not worth the time now. The machine has been brought back online and is working as expected. Lets cross fingers that this helped. But feel free to reopen if it doesn't.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Something is still broken here with accessing the external server for downloading the builds: urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> It needs further investigation. I will check that now.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Ok, so it looks like we really have to update the Win 8 VM! None of the proxy settings have been applied on that machine, so I assume it's also busted in the template. How can that happen? Any idea Adrian?
Flags: needinfo?(afernandez)
FYI, I fixed it on that machine, so only the template needs an update.
Seems the template did indeed not have the environmental proxy values (now it does). If you ever plan on doing any mass deployments of existing templates, let us know and we'll double check (run updates) the templates before deploying.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Flags: needinfo?(afernandez)
Resolution: --- → FIXED
Thanks Adrian! I will do this soon myself when I have to update the Windows templates anyway.
(In reply to Adrian Fernandez [:Aj] from comment #11) > Seems the template did indeed not have the environmental proxy values (now > it does). > > If you ever plan on doing any mass deployments of existing templates, let us > know and we'll double check (run updates) the templates before deploying. You missed to set the https_proxy environment variable on the mm-win-8-32-3 node. Can you please check if that hasn't been forgotten in the template too? Also this step is missing in the Mana page. So it has to be updated with the correct steps for environment variables.
Status: RESOLVED → REOPENED
Flags: needinfo?(afernandez)
Resolution: FIXED → ---
The Win_8_x86 template has the correct settings (system environmental proxy variables are set). I also verified that all the Windows templates had the correct environmental proxy variables set (I left them on for Bug 911452)
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Flags: needinfo?(afernandez)
Resolution: --- → FIXED
Looks good. Thanks Adrian.
Status: RESOLVED → VERIFIED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.