389703 - Use a proxy server to serve pages for the Talos testsuite

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rob Arnold [:robarnold]]

Attachment: Proposed solution

Using a proxy instead of mucking with each page is a cleaner solution.

Comment 1

[Rob Arnold [:robarnold]]

Attachment: Revised proxy server

Made the code easier to integrate with other python programs.
Fixed an issue where sites that used transfer-encoding: chunked wouldn't load in fx2.

Example embedding case:
from proxy import *
configure_proxy(localonly=True,port=8000)
run_proxy()

Comment 2

[Rob Arnold [:robarnold]]

Attachment: Fixed bug with options

Thanks to stuart for finding this

Comment 3

[Johnathan Nightingale [:johnath]]

While using the StandaloneTalos instructions here ( http://wiki.mozilla.org/StandaloneTalos )to build my own page set (step 5) I get frequent failures where the proxy reports connection problems, followed by my run_tests setup session failing.  qq.com and wretch.cc cause it somewhat reproducibly, but it just happened again with fotolog.net, errors pasted below.  I haven't yet gotten through a successful page set session.

Alice asked me to comment here, and include error output.

The output in the page set building window is:

C:\talos>python run_tests.py standalone\setup.config
Running test: tp_js
        Screen width/height:1920/1200
        colorDepth:24
        Browser inner width/height: 1006/615
        Browser outer width/height: 1024/768
FAIL: timeout:http://www.fotolog.net
FAIL: failure to complete test: tp_js



In the proxy server window, I get:

C:\talos>python standalone\proxyserver.py -p 9001
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58412)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58664)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58665)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58663)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58704)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58799)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58815)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58884)
Traceback (most recent call last):
  File "C:\Python25\lib\SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python25\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python25\lib\SocketServer.py", line 522, in __init__
    self.handle()
  File "C:\Python25\lib\BaseHTTPServer.py", line 318, in handle
    self.handle_one_request()
  File "C:\Python25\lib\BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "C:\Python25\lib\socket.py", line 346, in readline
    data = self._sock.recv(self._rbufsize)
error: (10053, 'Software caused connection abort')
----------------------------------------

Comment 4

[alice nodelman [:alice] [:anode]]

I'd like to land the proxy server in talos/tools, unless there are any objections.

Comment 5

[Rob Campbell [:rc] (:robcee)]

that's going to be somewhat deep and as I understand it, the proxy server can be used standalone. I'd recommend testing/proxyserver to minimize nesting.

Comment 6

[Rob Campbell [:rc] (:robcee)]

or testing/tools/proxyserver so it can be next to its friends the grabber and pageloader...

Comment 7

[[e-mail] (:bhearsum)]

testing/proxyserver (or testing/tools/proxyserver as was suggested on IRC) both sound fine. I don't think it should be inside of Talos; most of our Talos machines don't need it at all.

Comment 8

[alice nodelman [:alice] [:anode]]

Checking in proxyserver/proxyserver.py;
/cvsroot/mozilla/testing/tools/proxyserver/proxyserver.py,v  <--  proxyserver.py
initial revision: 1.1
done

Comment 9

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:Release Engineering:Talos. Filter on RelEngTalosMassMove to ignore.