Closed Bug 389703 Opened 14 years ago Closed 13 years ago

Use a proxy server to serve pages for the Talos testsuite

Categories

(Release Engineering :: General, defect)

x86
All
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: robarnold, Assigned: robarnold)

References

Details

Attachments

(1 file, 2 obsolete files)

Attached file Proposed solution (obsolete) —
Using a proxy instead of mucking with each page is a cleaner solution.
Blocks: 386081
Attached file Revised proxy server (obsolete) —
Made the code easier to integrate with other python programs.
Fixed an issue where sites that used transfer-encoding: chunked wouldn't load in fx2.

Example embedding case:
from proxy import *
configure_proxy(localonly=True,port=8000)
run_proxy()
Attachment #274007 - Attachment is obsolete: true
Attached file Fixed bug with options
Thanks to stuart for finding this
Attachment #277753 - Attachment is obsolete: true
While using the StandaloneTalos instructions here ( http://wiki.mozilla.org/StandaloneTalos )to build my own page set (step 5) I get frequent failures where the proxy reports connection problems, followed by my run_tests setup session failing.  qq.com and wretch.cc cause it somewhat reproducibly, but it just happened again with fotolog.net, errors pasted below.  I haven't yet gotten through a successful page set session.

Alice asked me to comment here, and include error output.

The output in the page set building window is:

C:\talos>python run_tests.py standalone\setup.config
Running test: tp_js
        Screen width/height:1920/1200
        colorDepth:24
        Browser inner width/height: 1006/615
        Browser outer width/height: 1024/768
FAIL: timeout:http://www.fotolog.net
FAIL: failure to complete test: tp_js



In the proxy server window, I get:

C:\talos>python standalone\proxyserver.py -p 9001
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58412)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58664)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58665)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58663)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58704)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58799)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58815)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58884)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------

I'd like to land the proxy server in talos/tools, unless there are any objections.
that's going to be somewhat deep and as I understand it, the proxy server can be used standalone. I'd recommend testing/proxyserver to minimize nesting.
or testing/tools/proxyserver so it can be next to its friends the grabber and pageloader...
testing/proxyserver (or testing/tools/proxyserver as was suggested on IRC) both sound fine. I don't think it should be inside of Talos; most of our Talos machines don't need it at all.
Checking in proxyserver/proxyserver.py;
/cvsroot/mozilla/testing/tools/proxyserver/proxyserver.py,v  <--  proxyserver.py
initial revision: 1.1
done
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Mass move of Core:Testing bugs to mozilla.org:Release Engineering:Talos. Filter on RelEngTalosMassMove to ignore.
Component: Testing → Release Engineering: Talos
Product: Core → mozilla.org
QA Contact: testing → release
Version: unspecified → other
Component: Release Engineering: Talos → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.