Closed Bug 915563 Opened 11 years ago Closed 9 years ago

Testrun stopped with "socket.error: [Errno 10022] WSAEINVAL"

Categories

(Testing Graveyard :: Mozmill, defect)

All
Windows Vista
defect
Not set
critical

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: cosmin-malutan, Unassigned)

References

Details

Attachments

(1 file)

Testrun stopped  with message "socket.error: [Errno 10022] WSAEINVAL" on Windows Vista x86 

Here is Jenkins log:

TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | setupModule
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | test1.js::setupModule
TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | testOpenAndCloseAddonManager
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | test1.js::testOpenAndCloseAddonManager
TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | teardownModule
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testAddons_OpenAndCloseGetAddonsPane\test1.js | test1.js::teardownModule
TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | setupModule
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | test1.js::setupModule
TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | testNavigateBackForward
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | test1.js::testNavigateBackForward
TEST-START | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | teardownModule
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests\tests\endurance\testBackForwardNavigation\test1.js | test1.js::teardownModule
[Errno 10022] WSAEINVAL
*** Uninstalling c:\users\mozauto\appdata\local\temp\tmp49g4wx.binary\
*** Removing repository 'c:\users\mozauto\appdata\local\temp\tmphpx87f.mozmill-tests'
Traceback (most recent call last):
  File "mozmill-automation/testrun_endurance.py", line 52, in <module>
    main()
  File "mozmill-automation/testrun_endurance.py", line 47, in main
    EnduranceTestRun().run()
  File "c:\jenkins\workspace\mozilla-central_endurance\mozmill-automation\libs\testrun.py", line 439, in run
    raise self.last_exception
socket.error: [Errno 10022] WSAEINVAL
Build step 'Invoke XShell command' marked build as failure
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Sending email to: mozmill-ci@mozilla.org
Finished: FAILURE
Happened again Windows Vista x86 with Aurora
Machine  mm-win-vista-32-1 

Jenkins Log:

TEST-START | c:\users\mozauto\appdata\local\temp\tmprqcbvn.mozmill-tests\tests\endurance\testTabbedBrowsing_OpenNewTab\test1.js | teardownModule
TEST-PASS | c:\users\mozauto\appdata\local\temp\tmprqcbvn.mozmill-tests\tests\endurance\testTabbedBrowsing_OpenNewTab\test1.js | test1.js::teardownModule
[Errno 10022] WSAEINVAL
*** Uninstalling c:\users\mozauto\appdata\local\temp\tmpucg13w.binary\
*** Removing repository 'c:\users\mozauto\appdata\local\temp\tmprqcbvn.mozmill-tests'
Traceback (most recent call last):
  File "mozmill-automation/testrun_endurance.py", line 52, in <module>
    main()
  File "mozmill-automation/testrun_endurance.py", line 47, in main
    EnduranceTestRun().run()
  File "c:\jenkins\workspace\mozilla-aurora_endurance\mozmill-automation\libs\testrun.py", line 439, in run
    raise self.last_exception
socket.error: [Errno 10022] WSAEINVAL
Build step 'Invoke XShell command' marked build as failure
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Sending email to: mozmill-ci@mozilla.org
Finished: FAILURE
This is not a mozmill-tests but a framework issue. Also given that we get killed it's a critical bug. It doesn't happen that often and we don't know how it correlates to Mozmill 2.0. So for now it's not blocking the next releases.

Cosmin, can you please check on which machines that happens? Is it always mm-win-vista-32-1?
Severity: normal → critical
Component: Mozmill Tests → Mozmill
Product: Mozilla QA → Testing
Summary: [Mozmill] Testrun stopped with "socket.error: [Errno 10022] WSAEINVAL" → Testrun stopped with "socket.error: [Errno 10022] WSAEINVAL"
It happened again on mm-win-vista-32-1. I did a quick look at this machine and 2 Firefox stopped dialogs were open. So we crashed with Nightly and Aurora yesterday. It might have an influence. I closed both dialogs now. 

The crashing information is:
Faulting application firefox.exe, version 26.0.0.5003, time stamp 0x5231bc6d, faulting module xul.dll, version 26.0.0.5003, time stamp 0x5231bc0c, exception code 0xc0000005, fault offset 0x0019daba, process id 0xbe0, application start time 0x01ceafd33142bd0a.
Otilia, why is no-one taking care of this bug while it is failing nearly all the time on the Windows Vista machine since a couple of days? This should have been caught by any of the test duties in that time frame of the last two weeks.
Flags: needinfo?(otilia.anica)
We keep focus on our top bugs and this bug is more infrastructure related, not mozmill tests. 
Cosmin may have a look on it later, once machine mm-win-vista-32-1 is free, as we've seen the failures only on this machine so far.
Flags: needinfo?(otilia.anica)
We had another failured on the same mm-win-vista-32-1 machine with a different console error message. Figured they could be related or could help the investigation:

http://mozmill-daily.blargon7.com/#/endurance/report/6ec6776efe900da3fd2b64a7506e9d64

Jenkins console log:
05:49:46 ###!!! [Parent][RPCChannel] Error: Channel timeout: cannot send/recv
05:49:46 
05:49:55 Timeout: bridge.execFunction("6eab1161-2a97-11e3-b0b3-005056bb7a66", bridge.registry["{5ab320b8-9939-4f82-afd0-2ddabb5584c3}"]["runTestFile"], ["c:\\users\\mozauto\\appdata\\local\\temp\\tmpyf4gja.mozmill-tests\\tests\\endurance\\testFlash_SWFVideoURL\\test1.js"])
05:49:55 
05:49:55 TEST-UNEXPECTED-FAIL | Disconnect Error: Application unexpectedly closed
I saw this failure twice today on mm-win-vista-32-1 Aurora and Nightly, I will take the machine offline  and try to reproduce.
I couldn't reproduced the failure and did't find anything on Events Viewer.
I investigated this today, It reproduced twice when I runned endurance tests.
I couldn't reduce the test but will keep running endurance and try to reproduce it when the machine won't be busy.
Assignee: nobody → cosmin.malutan
What exactly happened here? A hang, crash?
None, FF closed just fine and the testrun stopped.
Do we have any entry in the Windows Event viewer? This might give us some details.
Hi Henrik sorry for late response, as I said before on comment 8, I had seen no errors on event viewer.

Yesterday it failed on mm-win-8-64-3,  during Aurora update testrun.
After it failed I runned about 20 update tests and it didn't reproduced, I also runned endurance tests  on other vista machines that vista-32-1 but still didn't reproduced. It fails very rare. The only time I reproduced this was when I connected to the affected machine (mm-win-vista-32-1) right away after failure.
This failed on a Windows 8 box: mm-win-8-64-3
We are not confined to mm-win-vista-32-1

This was during an update testrun:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-aurora_update/4765/

Interestingly it failed _after_ the DirectDownload testrun:
> 04:44:32 INFO Passed: 8
> 04:44:32 INFO Failed: 0
> 04:44:32 INFO Skipped: 0
> 04:44:33 Report document created at 'http://mozauto.iriscouch.com/mozmill-daily/f54ca65f73fd56845c30b7ab7dbb9e63'
> 04:44:33 *** Removing updates staging folder 'C:\Users\mozauto\AppData\Local\Mozilla\updates\50ED85EB0F0E6CC1\updates\0'
> 04:44:33 *** Uninstalling c:\users\mozauto\appdata\local\temp\tmpo0nbe8.binary\
> 04:44:34 *** Removing test repository 'c:\users\mozauto\appdata\local\temp\tmp5hngw0.mozmill-tests'
> 04:44:35 Traceback (most recent call last):
> 04:44:35   File "c:\jenkins\workspace\mozilla-aurora_update\mozmill-env\python\Scripts\testrun_update-script.py", line 8, in <module>
> 04:44:35     load_entry_point('mozmill-automation==1.5.24', 'console_scripts', 'testrun_update')()
> 04:44:35   File "c:\jenkins\workspace\mozilla-aurora_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 1024, in update_cli
> 04:44:35     exec_testrun(UpdateTestRun)
> 04:44:35   File "c:\jenkins\workspace\mozilla-aurora_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 998, in exec_testrun
> 04:44:35     cls().run()
> 04:44:35   File "c:\jenkins\workspace\mozilla-aurora_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 982, in run
> 04:44:35     TestRun.run(self)
> 04:44:35   File "c:\jenkins\workspace\mozilla-aurora_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 449, in run
> 04:44:35     raise self.last_exception
> 04:44:35 socket.error: [Errno 10022] WSAEINVAL
> 04:44:35 Build step 'Invoke XShell command' marked build as failure
> 04:44:35 Archiving artifacts
> 04:44:35 Recording test results
> 04:46:13 Email was triggered for: Failure
> 04:46:13 Sending email for trigger: Failure
> 04:46:27 Sending email to: mozmill-ci@mozilla.org
> 04:46:27 Finished: FAILURE
The bad thing is that we are hiding the real line of code for those exceptions. It should get better with the hotfix-2.0 branch. So we might know where exactly we are failing.
I'm not that this is good or bad news.
Another WSAEINVAL failure, this time on mm-win-xp-32-3:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-esr17_functional/1229/

> 04:26:13 TEST-START | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_enableDisableExtension\test4.js | teardownModule
> 04:26:13 TEST-PASS | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_enableDisableExtension\test4.js | test4.js::teardownModule
> 04:26:20 TEST-START | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | setupModule
> 04:26:20 TEST-PASS | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | test1.js::setupModule
> 04:26:20 TEST-START | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | testInstallMultipleExtensions
> 04:26:25 TEST-PASS | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | test1.js::testInstallMultipleExtensions
> 04:26:25 TEST-START | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | teardownModule
> 04:26:25 TEST-PASS | c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests\tests\functional\restartTests\testAddons_installMultipleExtensions\test1.js | test1.js::teardownModule
> 04:26:31 [Errno 10022] WSAEINVAL
> 04:26:31 *** Uninstalling c:\docume~1\mozilla\locals~1\temp\tmped898t.binary\
> 04:26:32 *** Removing test repository 'c:\docume~1\mozilla\locals~1\temp\tmpupncxv.mozmill-tests'
> 04:26:36 Traceback (most recent call last):
> 04:26:36   File "c:\jenkins\workspace\mozilla-esr17_functional\mozmill-env\python\Scripts\testrun_functional-script.py", line 8, in <module>
> 04:26:36     load_entry_point('mozmill-automation==1.5.24', 'console_scripts', 'testrun_functional')()
> 04:26:36   File "c:\jenkins\workspace\mozilla-esr17_functional\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 1012, in functional_cli
> 04:26:36     exec_testrun(FunctionalTestRun)
> 04:26:36   File "c:\jenkins\workspace\mozilla-esr17_functional\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 998, in exec_testrun
> 04:26:36     cls().run()
> 04:26:36   File "c:\jenkins\workspace\mozilla-esr17_functional\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 449, in run
> 04:26:36     raise self.last_exception
> 04:26:36 socket.error: [Errno 10022] WSAEINVAL
> 04:26:36 Build step 'Invoke XShell command' marked build as failure
> 04:26:36 Archiving artifacts
> 04:26:36 Recording test results
> 04:26:36 Email was triggered for: Failure
> 04:26:36 Sending email for trigger: Failure
> 04:26:37 Sending email to: mozmill-ci@mozilla.org
> 04:26:37 Finished: FAILURE
Strangely Windows Defender was still active on the vista-32-1 machine. I disabled it now. Lets see if that makes a change here.
Not sure what happened with those machines but also the screensaver was enabled after 10min. So it might have had also an effect.
This happened on mm-win-8-32-1 last night with ondemand update tests

http://mozmill-release.blargon7.com/#/update/report/8a42e4b8f277f9c0268b061eb8424066
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/ondemand_update/32184/

13:24:35 *** Removing updates staging folder 'C:\Users\mozauto\AppData\Local\Mozilla\updates\55752D4D03350100\updates\0'
13:24:35 *** Uninstalling c:\users\mozauto\appdata\local\temp\tmpddkdus.binary\
13:24:38 *** Removing test repository 'c:\users\mozauto\appdata\local\temp\tmpk0e5a5.mozmill-tests'
13:24:38 Traceback (most recent call last):
13:24:38   File "c:\jenkins\workspace\ondemand_update\mozmill-env\python\Scripts\testrun_update-script.py", line 8, in <module>
13:24:38     load_entry_point('mozmill-automation==1.5.24', 'console_scripts', 'testrun_update')()
13:24:38   File "c:\jenkins\workspace\ondemand_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 1024, in update_cli
13:24:38     exec_testrun(UpdateTestRun)
13:24:38   File "c:\jenkins\workspace\ondemand_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 998, in exec_testrun
13:24:38     cls().run()
13:24:38   File "c:\jenkins\workspace\ondemand_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 982, in run
13:24:38     TestRun.run(self)
13:24:38   File "c:\jenkins\workspace\ondemand_update\mozmill-env\python\lib\site-packages\mozmill_automation\testrun.py", line 449, in run
13:24:38     raise self.last_exception
13:24:38 socket.error: [Errno 10022] WSAEINVAL
13:24:39 Build step 'Invoke XShell command' marked build as failure
13:24:39 Archiving artifacts
13:24:39 Recording test results
13:25:52 Email was triggered for: Failure
13:25:52 Sending email for trigger: Failure
13:25:55 Sending email to: mozmill-ci@mozilla.org
13:25:55 Finished: FAILURE
Happened again on Windows xp x86 with Nightly en-US during the update tests 
Machine mm-win-xp-32-1.
The actual machine is  mm-win-8-64-4, I take it offline to try to reproduce the failure.
We haven't seen this for a while now.
It did crash yesterday evening on mm-win-8-32-3

As pointed out in comment 17 I'll check the machine for Windows Defender and/or screensaver being active.
This fails very rarely, it does not reproduce anymore even if we connect right away on the affected nodes. Whatever was the cause of this is gone. If it will fail again I will try to investigate further but for now I close this bug.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
This was seen just two days ago on http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-esr24_update/755/console as reported by Andrei.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Attached is the console log for the most recently reported occurrence.
I investigated this today and it seems that we get a WSAEINVAL when we try to send the report:
Here it says that WSAEINVAL is raised when we try to connect to the same host with the same socket after the connection was refused:
http://www.gossamer-threads.com/lists/python/python/42067

http://msdn.microsoft.com/en-us/library/windows/desktop/ms740668(v=vs.85).aspx#WSAEINVAL
Interesting. Then it would be good to know then if another process on that machine has already opened a socket to this host. I would suspect a zombie process of the testrun script. Once this failure happens again, can you make sure to check such a machine?
Hi,
It just failed again on mm-win-xp-32-2  I checked for python processes under the task manager, as for I know that is the testrun process. There were no zombie processes.
This bug should have been fixed by the landing of my patch on bug 956315. We shouldn't see it anymore with Mozmill 2.0.4 and later.
Assignee: cosmin.malutan → hskupin
Status: REOPENED → RESOLVED
Closed: 11 years ago10 years ago
Hardware: x86 → All
Resolution: --- → FIXED
Whiteboard: [mozmill-2.0.4]
This happened again today:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-aurora_functional/18987/console

05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\mozmill\__init__.py", line 287, in create_network
05:18:04     self.jsbridge_port)
05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\__init__.py", line 47, in wait_and_create_network
05:18:04     back_channel, bridge = create_network(host, port)
05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 295, in create_network
05:18:04     back_channel = BackChannel(hostname, port)
05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 250, in __init__
05:18:04     Bridge.__init__(self, host, port)
05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 151, in __init__
05:18:04     self.connect((host, port))
05:18:04   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\Lib\asyncore.py", line 353, in connect
05:18:04     raise socket.error(err, errorcode[err])
05:18:04 error: [Errno 10022] WSAEINVALBuild step 'Invoke XShell command' marked build as failure
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [mozmill-2.0.4]
This failed today for about 5 times on: 
mm-win-vista-32-2
mm-win-vista-32-3
mm-win-vista-32-4

>5:08:45     frame = self.run_test_file(frame or self.start_runner(),
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\mozmill\__init__.py", line 327, in start_runner
>05:08:45     self.create_network()
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\mozmill\__init__.py", line 287, in create_network
>05:08:45     self.jsbridge_port)
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\__init__.py", line 47, in wait_and_create_network
>05:08:45     back_channel, bridge = create_network(host, port)
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 295, in create_network
>05:08:45     back_channel = BackChannel(hostname, port)
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 250, in __init__
>05:08:45     Bridge.__init__(self, host, port)
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 151, in __init__
>05:08:45     self.connect((host, port))
>05:08:45   File "c:\jenkins\workspace\mozilla-aurora_functional\mozmill-env-windows\python\Lib\asyncore.py", line 353, in connect
>05:08:45     raise socket.error(err, errorcode[err])
>05:08:45 error: [Errno 10022] WSAEINVAL
>05:08:45 Build step 'Invoke XShell command' marked build as failure
It failed to connect to js bridge extension on the first unskiped test.
I ran this for 20 times on mm-win-vista-32-4 but it didn't reproduced.
Failed again on mm-win-xp-32-2, also before to start to run any test, when creating the socket for bridge:
bridge = create_network(host, port)
I took the node offline right away but I couldn't reproduce it in a testrun.

Here is the occurrence of this failure in the past month, by far it failed the most on mm-win-vista-32-4:

mm-win-vista-32-4 * 7
mm-win-xp-32-2    * 3
mm-win-vista-32-2 * 2
mm-win-vista-32-3 * 1 
mm-win-vista-32-1 * 1
mm-win-xp-32-1    * 1
Failed again with Nightly fr on Windows xp 32 (mm-win-xp-32-4)

File "c:\jenkins\workspace\mozilla-central_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 295, in create_network
06:56:41     back_channel = BackChannel(hostname, port)
06:56:41   File "c:\jenkins\workspace\mozilla-central_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 250, in __init__
06:56:41     Bridge.__init__(self, host, port)
06:56:41   File "c:\jenkins\workspace\mozilla-central_functional\mozmill-env-windows\python\lib\site-packages\jsbridge\network.py", line 151, in __init__
06:56:41     self.connect((host, port))
06:56:41   File "c:\jenkins\workspace\mozilla-central_functional\mozmill-env-windows\python\Lib\asyncore.py", line 353, in connect
06:56:41     raise socket.error(err, errorcode[err])
06:56:41 error: [Errno 10022] WSAEINVAL
We may wanna get this fixed for Mozmill 2.1. Adding to the dependency list.
Assignee: hskupin → nobody
Blocks: 970594
Status: REOPENED → NEW
We are not going back to use Mozmill for our UI testing of Firefox and this is not a real blocker for a possible release given its low appearance. Also it hasn't been seen for a while anymore.
Status: NEW → RESOLVED
Closed: 10 years ago9 years ago
Resolution: --- → WORKSFORME
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: