Closed Bug 546640 Opened 15 years ago Closed 14 years ago

Even without an internet connection jsbridge should be able to create localhost connections

Categories

(Testing Graveyard :: Mozmill, defect)

defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: standard8, Assigned: harth)

References

Details

(Whiteboard: [mozmill-1.4.2-][mozmill-1.5.2-][mozmill-2.0+])

Attachments

(2 files)

Attempting to use MozMill for automated testing in offline mode doesn't work. As soon as offline mode is dropped, jsbridge stops working. This means we can't test offline mode in Thunderbird. As Thunderbird does quite a few things differently in offline mode, being able to test that mode would be extremely useful. I suspect there is a core bug affecting us (i.e. drop all network connections in offline mode, not just local/select ones), though I don't know if MozMill may also be able to work around this in some manner.
Blocks: 538810
Mark, which errors are listed? Would be nice to know those because it looks like that this bug is eventually special for Thunderbird.
We get errors where the test harness attempts to make a network connection, which necko refuses. Even localhost connections are considered network connections by necko, I believe.
David, can you please c&p those errors here? That would be great. I would have to find a way to make it reproducible with Firefox's offline mode. Then we could ask our network gurus.
(In reply to comment #3) > David, can you please c&p those errors here? That would be great. I would have > to find a way to make it reproducible with Firefox's offline mode. Then we > could ask our network gurus. "Application unexpectedly closed" Just stick this bit of code in a mozmill test to reproduce: let ioService = Components.classes["@mozilla.org/network/io-service;1"] .getService(Components.interfaces.nsIIOService2); ioService.offline = true; This, of course, is effectively breaking the network connection between mozrunner and jsbridge as that's what going offline forces connections to do.
Mark, how important or offline mode tests for the Thunderbird team?
I would say this is an underlying network detection issue. I have tried to use toolkit.networkmanager.disable=true but it doesn't work during startup. In that case Firefox gets started always in offline mode when no internet connection is present. Then we can't connect to localhost. See also bug 339814. Boris, is there a preference we can use to completely disable the automatic offline detection?
Depends on: 339814
Whiteboard: [mozmill-2.0?]
Summary: MozMill/jsbridge/Mozrunner doesn't work in offline mode → Offline mode shouldn't prevent jsbridge from creating localhost connections
Whiteboard: [mozmill-2.0?] → [mozmill-1.4.2?]
Attached patch Patch v1Splinter Review
With the help of biesi I was able to circumvent this problem until the underlying networking bug has been fixed. With network.manage-offline-status set to false per default and forcing to stay in online mode at start-up we can make sure that we will always be able to connect to localhost. If a test needs the offline mode we should file a new bug to implement listeners for.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Attachment #456980 - Flags: review?
Attachment #456980 - Flags: review? → review?(ctalbert)
Summary: Offline mode shouldn't prevent jsbridge from creating localhost connections → Even without an internet connection jsbridge should be able to create localhost connections
(In reply to comment #7) > With the help of biesi I was able to circumvent this problem until the > underlying networking bug has been fixed. With network.manage-offline-status > set to false per default and forcing to stay in online mode at start-up we can > make sure that we will always be able to connect to localhost. That wasn't what this bug was about (we force online mode in Thunderbird tests already in case your computer is offline, or if we don't we can do ;-) ). > If a test needs the offline mode we should file a new bug to implement > listeners for. That is what this bug is about - We want to be able to do tests where we can put the application into offline mode whilst running, and not have jsbridge disconnected (as per comment 4).
(In reply to comment #8) > That wasn't what this bug was about (we force online mode in Thunderbird tests > already in case your computer is offline, or if we don't we can do ;-) ). For the future I would love it if we could backport those important changes into Mozmill itself. You guys have probably some more updates which could help us too. > That is what this bug is about - We want to be able to do tests where we can > put the application into offline mode whilst running, and not have jsbridge > disconnected (as per comment 4). Sorry for hijacking the bug then. But as long as bug 339814 isn't fixed we probably can't do anything on that front. No idea if it's really worth filing a new bug now. IMO your offline tests should work once bug 339814 has been fixed. But feel free to create one or comment on the other bug.
(In reply to comment #9) > (In reply to comment #8) > > That wasn't what this bug was about (we force online mode in Thunderbird tests > > already in case your computer is offline, or if we don't we can do ;-) ). > > For the future I would love it if we could backport those important changes > into Mozmill itself. You guys have probably some more updates which could help > us too. Well, that's just a pref that we set in our default profile: http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtest.py#164 Unfortunately, "offline.autoDetect" pref that applies to mailnews apps only. Pretty much everything we do in that runtest.py is profile set up and managing results. We don't actually modify mozmill at all (although we do have our own shared-modules). Last I heard Clint & co were thinking of running mozmill in a different way to our scripts, so I'm not sure there is much to back-port.
(In reply to comment #10) > Well, that's just a pref that we set in our default profile: > > http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtest.py#164 > > Unfortunately, "offline.autoDetect" pref that applies to mailnews apps only. That means you do not need the network.manage-offline-status pref set to false in the Thunderbird profile? Shall I remove it from the patch? > Pretty much everything we do in that runtest.py is profile set up and managing > results. We don't actually modify mozmill at all (although we do have our own > shared-modules). Last I heard Clint & co were thinking of running mozmill in a > different way to our scripts, so I'm not sure there is much to back-port. Not for this but, that's correct.
Comment on attachment 456980 [details] [diff] [review] Patch v1 This doesn't solve the problem this bug is trying to solve. I think what we really need to look into here in order to solve this is to call out to a non-necko library to do the networking that jsbridge needs then jsbridge will not be dependent on whether necko is "online" or "offline". Then we can use that library for our network connection and be able to test both online and offline capabilties. We need: * c library to open a local socket * a way to communicate to that socket from js to make the communication seamless from the perspective of jsbridge (i.e. a layer that calls out to the c library via ctypes) * Tests to ensure all this works * Builds for each target OS. This really just obscures the issue, and for that reason I'm against checking it in unless there really is a pressing need. As far as I can see, this is something that any old test could set in their startup profile and so I'd rather handle this there at that level so that we are explicitly forcing the tests to make this decision than hacking around something that we should fix.
Attachment #456980 - Flags: review?(ctalbert) → review-
Whiteboard: [mozmill-1.4.2?] → [mozmill-1.4.2-]
(In reply to comment #12) > * a way to communicate to that socket from js to make the communication > seamless from the perspective of jsbridge (i.e. a layer that calls out to the c > library via ctypes) ctypes is not supported on 1.9.2 and 1.9.1. Please keep this in mind when you start implementing that. Looks like we need binary platform dependent components.
Whiteboard: [mozmill-1.4.2-] → [mozmill-1.4.2-][mozmill-1.4.3?]
Assignee: hskupin → nobody
Status: ASSIGNED → NEW
> * c library to open a local socket How about NSPR? > ctypes is not supported on 1.9.2 and 1.9.1. Please keep this in mind when you > start implementing that. Looks like we need binary platform dependent > components. I believe 1.9.2 has basic ctypes support, actually. Even if that support turns out not to be good enough, having a fix for this that required Gecko > 1.9.2 would still be worthwhile...
Presumably depending on NSPR also avoids us having to do build any separate per-platform component builds.
(In reply to comment #14) > I believe 1.9.2 has basic ctypes support, actually. Even if that support turns > out not to be good enough, having a fix for this that required Gecko > 1.9.2 > would still be worthwhile... You are right. It has also been landed on 1.9.2 (see bug 513783)
I'd love to use NSPR for this but as that is what we're doing to open the socket right now and the socket gets nixed when we go into offline mode, I don't know if that would work. Do you mean using NSPR via ctypes as a means to go around the "offline/online observer" mechanism that XPCOM is imposing on NSPR?
Yes, talking directly to NSPR rather than going through Necko is what I was trying to suggest.
Whiteboard: [mozmill-1.4.2-][mozmill-1.4.3?] → [mozmill-1.4.2-][mozmill-1.5.1?]
This patch uses js-ctypes to call the NSPR socket methods instead of using nsIServerSocket and nsISocketTransport that get shutdown during offline mode.
Assignee: nobody → fayearthur+bugs
Attachment #478339 - Flags: review?(ctalbert)
(In reply to comment #19) > This patch uses js-ctypes to call the NSPR socket methods instead of using > nsIServerSocket and nsISocketTransport that get shutdown during offline mode. Please be aware that js-ctypes is not supported on 1.9.1. As long as we support 1.9.1 we need a fallback solution.
(In reply to comment #20) > (In reply to comment #19) > > This patch uses js-ctypes to call the NSPR socket methods instead of using > > nsIServerSocket and nsISocketTransport that get shutdown during offline mode. > > Please be aware that js-ctypes is not supported on 1.9.1. As long as we support > 1.9.1 we need a fallback solution. yeah, this will only work on Firefox 4+. It uses the old stuff otherwise.
Whiteboard: [mozmill-1.4.2-][mozmill-1.5.1?] → [mozmill-1.4.2-][mozmill-1.5.2?]
Comment on attachment 478339 [details] [diff] [review] use NSPR sockets directly This looks really good. I tested it a bit on windows and I like the way we fall back to the old method in case anything goes wrong. I wish there were a way to write some message to the log system for Mozmill in the case that we can't instantiate nspr sockets so that it would be easier to realize that we are falling back, but because we ARE the mechanism that writes such messages, that's a unique chicken and egg problem. Can we write a message to the Mozmill logging system if we startup the old way that says "COULD NOT INSTANTIATE NSPR - ALL OFFLINE TESTS WILL FAIL". That way we can detect this failure? You also need a license block on nspr.js and nspr-socket.js. Let's also take this opportunity to clean up those other license headers so that they have the standard formatting while we're in here moving all this code around. r+ with all these changes. >diff --git a/jsbridge/jsbridge/extension/components/cmdarg.js b/jsbridge/jsbridge/extension/components/cmdarg.js >index 6623721..87ebec9 100644 >diff --git a/jsbridge/jsbridge/extension/resource/modules/bridge.js b/jsbridge/jsbridge/extension/resource/modules/bridge.js >new file mode 100644 >index 0000000..69cc637 >--- /dev/null >+++ b/jsbridge/jsbridge/extension/resource/modules/bridge.js >@@ -0,0 +1,156 @@ >+// ***** BEGIN LICENSE BLOCK ***** >+// Version: MPL 1.1/GPL 2.0/LGPL 2.1 >+// >+// The contents of this file are subject to the Mozilla Public License Version >+// 1.1 (the "License"); you may not use this file except in compliance with >+// the License. You may obtain a copy of the License at >+// http://www.mozilla.org/MPL/ >+// >+// Software distributed under the License is distributed on an "AS IS" basis, >+// WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License >+// for the specific language governing rights and limitations under the >+// License. >+// >+// The Original Code is Mozilla Corporation Code. >+// >+// The Initial Developer of the Original Code is >+// based on the MozRepl project. >+// Portions created by the Initial Developer are Copyright (C) 2008 >+// the Initial Developer. All Rights Reserved. >+// >+// Contributor(s): >+// Mikeal Rogers <mikeal.rogers@gmail.com> >+// Massimiliano Mirra <bard@hyperstruct.net> >+// >+// Alternatively, the contents of this file may be used under the terms of >+// either the GNU General Public License Version 2 or later (the "GPL"), or >+// the GNU Lesser General Public License Version 2.1 or later (the "LGPL"), >+// in which case the provisions of the GPL or the LGPL are applicable instead >+// of those above. If you wish to allow use of your version of this file only >+// under the terms of either the GPL or the LGPL, and not to allow others to >+// use your version of this file under the terms of the MPL, indicate your >+// decision by deleting the provisions above and replace them with the notice >+// and other provisions required by the GPL or the LGPL. If you do not delete >+// the provisions above, a recipient may use your version of this file under >+// the terms of any one of the MPL, the GPL or the LGPL. >+// >+// ***** END LICENSE BLOCK ***** Can we change these license blocks so that they use the standard license block comments - see http://www.mozilla.org/MPL/boilerplate-1.1/mpl-tri-license-c. The content of the block is ok, but since we're moving this code around, let's switch these to use the * comments the way they should have been done in the first place. >+ if (data != undefined) { >+ this.set(uuid, data); >+ } else if ( result == true) { >+ this.session.encodeOut({'result':true, 'data':null, 'uuid':uuid}); >+ } else { >+ throw 'Something very bad happened.' >+ } Can we make this something more grep-able? Like: throw 'JSBridge ERROR: Invalid function call into bridge' >+++ b/jsbridge/jsbridge/extension/resource/modules/nspr-server.js >@@ -0,0 +1,143 @@ >+// ***** BEGIN LICENSE BLOCK ***** >+// Version: MPL 1.1/GPL 2.0/LGPL 2.1 >+// >+// The contents of this file are subject to the Mozilla Public License Version >+// 1.1 (the "License"); you may not use this file except in compliance with >+// the License. You may obtain a copy of the License at >+// http://www.mozilla.org/MPL/ >+// >+// Software distributed under the License is distributed on an "AS IS" basis, >+// WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License >+// for the specific language governing rights and limitations under the >+// License. >+// >+// The Original Code is Mozilla Corporation Code. >+// >+// The Initial Developer of the Original Code is >+// based on the MozRepl project. >+// Portions created by the Initial Developer are Copyright (C) 2008 >+// the Initial Developer. All Rights Reserved. >+// >+// Contributor(s): >+// Mikeal Rogers <mikeal.rogers@gmail.com> >+// Massimiliano Mirra <bard@hyperstruct.net> >+// >+// Alternatively, the contents of this file may be used under the terms of >+// either the GNU General Public License Version 2 or later (the "GPL"), or >+// the GNU Lesser General Public License Version 2.1 or later (the "LGPL"), >+// in which case the provisions of the GPL or the LGPL are applicable instead >+// of those above. If you wish to allow use of your version of this file only >+// under the terms of either the GPL or the LGPL, and not to allow others to >+// use your version of this file under the terms of the MPL, indicate your >+// decision by deleting the provisions above and replace them with the notice >+// and other provisions required by the GPL or the LGPL. If you do not delete >+// the provisions above, a recipient may use your version of this file under >+// the terms of any one of the MPL, the GPL or the LGPL. >+// >+// ***** END LICENSE BLOCK ***** Let's change this license block to use the * style comments too. >+var toUnicode = function(text, charset) { >+ var converter = Components.classes["@mozilla.org/intl/scriptableunicodeconverter"] >+ .createInstance(Components.interfaces.nsIScriptableUnicodeConverter); You've got a stray tab character on this line ^ >diff --git a/jsbridge/jsbridge/extension/resource/modules/nspr-socket.js b/jsbridge/jsbridge/extension/resource/modules/nspr-socket.js >new file mode 100644 >index 0000000..79a9e62 >--- /dev/null >+++ b/jsbridge/jsbridge/extension/resource/modules/nspr-socket.js >@@ -0,0 +1,110 @@ >+var nspr = {}; Components.utils.import("resource://jsbridge/modules/nspr.js", nspr); >+var nsprTypes = nspr.nsprTypes; >+nspr = nspr.nsprSockets; >+ >+var hwindow = Components.classes["@mozilla.org/appshell/appShellService;1"] >+ .getService(Components.interfaces.nsIAppShellService) >+ .hiddenDOMWindow; >+ >+var EXPORTED_SYMBOLS = ["ServerSocket"]; >+ >+var ServerSocket = function(port) { >+ var addr = nsprTypes.PRNetAddr(); >+ nspr.PR_SetNetAddr(nspr.PR_IpAddrLoopback, nspr.PR_AF_INET, >+ port, addr.address()); >+ >+ var fd = nspr.PR_OpenTCPSocket(nspr.PR_AF_INET); >+ >+ // don't block for accept/send/recv >+ var opt = nsprTypes.PRSocketOptionData(); >+ opt.non_blocking = nspr.PR_TRUE; >+ opt.option = nspr.PR_SockOpt_Nonblocking; >+ nspr.PR_SetSocketOption(fd, opt.address()); >+ >+ // don't buffer when sending >+ var opt = nsprTypes.PRSocketOptionData(); >+ opt.non_blocking = nspr.PR_TRUE; // same space >+ opt.option = nspr.PR_SockOpt_NoDelay; >+ nspr.PR_SetSocketOption(fd, opt.address()); >+ >+ // allow local address re-use >+ var opt = nsprTypes.PRSocketOptionData(); >+ opt.non_blocking = nspr.PR_TRUE; // same space >+ opt.option = nspr.PR_SockOpt_Reuseaddr; >+ nspr.PR_SetSocketOption(fd, opt.address()); >+ >+ var status = nspr.PR_Bind(fd, addr.address()); >+ if(status != 0) >+ throw "socket failed to bind, kill all firefox processes"; >+ >+ var status = nspr.PR_Listen(fd, -1); >+ if(status != 0) >+ throw "socket failed to listen"; >+ >+ this.addr = addr; >+ this.fd = fd; >+} >+ >+ServerSocket.prototype = { >+ onConnect : function(callback, interval) { >+ interval = interval || 2000; >+ var that = this; >+ (function accept() { >+ var newfd = nspr.PR_Accept(that.fd, that.addr.address(), nspr.PR_INTERVAL_NO_WAIT); >+ if(!newfd.isNull()) >+ callback(new Client(newfd)); >+ >+ hwindow.setTimeout(accept, interval); >+ })(); >+ }, >+ >+ close : function() { >+ return nspr.PR_Close(this.fd); >+ } >+} >+ >+ >+var Client = function(fd) { >+ this.fd = fd; >+} >+ >+Client.prototype = { >+ onMessage : function(callback, interval, bufsize) { >+ bufsize = bufsize || 4096; >+ interval = interval || 1000; // polling interval >+ var reinterval = interval; // next check after receiving data >+ >+ var that = this; >+ (function getMessage() { >+ var buffer = new nspr.buffer(bufsize); >+ var bytes = nspr.PR_Recv(that.fd, buffer, bufsize, 0, nspr.PR_INTERVAL_NO_WAIT); >+ if(bytes > 0) { >+ var message = buffer.readString(); >+ callback(message); >+ reinterval = 50; >+ } >+ else if(bytes == 0) { >+ if(that.handleDisconnect) >+ that.handleDisconnect(); >+ return; >+ } >+ >+ reinterval = Math.min(reinterval, interval); >+ hwindow.setTimeout(getMessage, reinterval); >+ reinterval *= 2; // logarithmically back off >+ })(); >+ }, >+ >+ onDisconnect : function(callback) { >+ this.handleDisconnect = callback; >+ }, >+ >+ sendMessage : function(message) { >+ var buffer = new nspr.buffer(message); >+ nspr.PR_Send(this.fd, buffer, message.length, 0, nspr.PR_INTERVAL_NO_WAIT); >+ }, >+ >+ close : function() { >+ return nspr.PR_Close(this.fd); >+ } >+} >diff --git a/jsbridge/jsbridge/extension/resource/modules/nspr.js b/jsbridge/jsbridge/extension/resource/modules/nspr.js This file needs a license block. >diff --git a/jsbridge/jsbridge/extension/resource/modules/server.js b/jsbridge/jsbridge/extension/resource/modules/server.js >index cb58075..1fbd0f0 100644 >--- a/jsbridge/jsbridge/extension/resource/modules/server.js >+++ b/jsbridge/jsbridge/extension/resource/modules/server.js >@@ -36,25 +36,15 @@ > // > // ***** END LICENSE BLOCK ***** Fix this license block too, please.
Attachment #478339 - Flags: review?(ctalbert) → review+
(In reply to comment #22) > that's a unique chicken and egg problem. Can we write a message to the Mozmill > logging system if we startup the old way that says "COULD NOT INSTANTIATE NSPR > - ALL OFFLINE TESTS WILL FAIL". That way we can detect this failure? Could we expose the availability of the NSPR connection as a property bind to the mozmill object? That way test modules can benefit from and run different test paths or even disable those, if NSPR is not available. Could we also run into problems with trunk and 1.9.2 builds by setting up the connection? If yes, such a proposed property would be better as having a global log message.
(In reply to comment #23) > (In reply to comment #22) > > that's a unique chicken and egg problem. Can we write a message to the Mozmill > > logging system if we startup the old way that says "COULD NOT INSTANTIATE NSPR > > - ALL OFFLINE TESTS WILL FAIL". That way we can detect this failure? > > Could we expose the availability of the NSPR connection as a property bind to > the mozmill object? That way test modules can benefit from and run different > test paths or even disable those, if NSPR is not available. What Gecko application are you testing that does not have NSPR available? If there is any issue with NSPR, it falls back to using standard XPCOM. I do not want to have properties for no reason and this is really not something any test should ever be configuring as it is core to the harness. > Could we also run into problems with trunk and 1.9.2 builds by setting up the > connection? If yes, such a proposed property would be better as having a global > log message. No, as I said, if there is any issue whatsoever, we fall back to using the old XPCOM method which has worked since Gecko 1.9.0.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Tested by deactivating my airport connection and starting Mozmill on the master branch with Minefield still fails: jsbridge: Exception: [Exception... "Component returned failure code: 0xc1f30001 (NS_ERROR_NOT_INITIALIZED) [nsIServerSocket.asyncListen]" nsresult: "0xc1f30001 (NS_ERROR_NOT_INITIALIZED)" location: "JS frame :: resource://jsbridge/modules/server.js :: anonymous :: line 280" data: no] Traceback (most recent call last): File "/Volumes/data/testing/envs/dev/bin/mozmill", line 8, in <module> load_entry_point('mozmill==1.4.2b1', 'console_scripts', 'mozmill')() File "/Volumes/data/build/tools/mozmill/mozmill/mozmill/__init__.py", line 810, in cli CLI().run() File "/Volumes/data/build/tools/mozmill/mozmill/mozmill/__init__.py", line 758, in run self.mozmill.start(runner=runner, profile=runner.profile) File "/Volumes/data/build/tools/mozmill/mozmill/mozmill/__init__.py", line 244, in start self.appinfo = self.get_appinfo(self.bridge) File "/Volumes/data/build/tools/mozmill/mozmill/mozmill/__init__.py", line 329, in get_appinfo mozmill = jsbridge.JSObject(bridge, mozmillModuleJs) File "/Volumes/data/testing/envs/dev/lib/python2.6/site-packages/jsbridge-2.4.0-py2.6.egg/jsbridge/jsobjects.py", line 76, in __init__ name = bridge.set(name)['data'] File "/Volumes/data/testing/envs/dev/lib/python2.6/site-packages/jsbridge-2.4.0-py2.6.egg/jsbridge/network.py", line 227, in set return self.run(_uuid, 'bridge.set('+', '.join([encoder.encode(_uuid), obj_name])+')') File "/Volumes/data/testing/envs/dev/lib/python2.6/site-packages/jsbridge-2.4.0-py2.6.egg/jsbridge/network.py", line 201, in run raise JSBridgeDisconnectError("Connected disconnected") jsbridge.network.JSBridgeDisconnectError: Connected disconnected
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to comment #26) > line 201, in run > raise JSBridgeDisconnectError("Connected disconnected") > jsbridge.network.JSBridgeDisconnectError: Connected disconnected Are you absolutely certain you're running the new code? That's exactly the error you'd get if you were running the old code, which makes me suspicious.
Yes, I'm. Sadly I'm not able to start a master build currently because of bug 616895. I will revisit this bug once bug 616895 has been fixed.
This is on master, I'm not clear why we would want to take this on 1.5.2. The solution relies on js-ctypes which I think are only available in Gecko 2.x platforms and higher, so this is not a general availability fix, and I don't see any reason on this bug that justifies the cost of backporting it. I recommend we minux and do not take it for 1.5.2.
Whiteboard: [mozmill-1.4.2-][mozmill-1.5.2?] → [mozmill-1.4.2-][mozmill-1.5.2-][mozmill-2.0+]
setting to fixed again, the error in comment 26 is clearly coming from the old ServerSocket code.
Status: REOPENED → RESOLVED
Closed: 14 years ago14 years ago
Resolution: --- → FIXED
Ok, now with bug 616895 fixed I can run Mozmill 2.0 again on OS X. And I can't see this problem anymore. But, we now have a crash: bp-53116cab-09bd-4c86-aa4b-124d72110114. I will file a new bug for it.
Alright, so I have checked this again with a Minefield build before the crash has been started and it works perfectly. It's kinda sad we can't use it for Firefox 3.6. Marking as verified fixed.
Status: RESOLVED → VERIFIED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: