Closed Bug 387619 Opened 17 years ago Closed 16 years ago

Stand-alone Talos

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: anodelman, Assigned: anodelman)

References

Details

Attachments

(2 files)

Make Talos runable as a stand alone application.  Initially this includes:

- Removing the dependency on the graph server.  You should be able to run the tests locally and have the data dropped to either console or csv files
- Be able to choose which tests to run instead of having all tests run on all builds

Other considerations
- remove the dependency on installing a web server by providing one.  This is not as necessary, but would streamline the running of the tests and reduce the overhead to running them
Blocks: 386081
Regarding removing the dependency on installing a server, xpcshell and MochiTest both use a simple HTTP server written in JS and available in the source tree <http://lxr.mozilla.org/mozilla/source/netwerk/test/httpserver/>.

Docs on the server are available on devmo <http://developer.mozilla.org/en/docs/HTTP_server_for_unit_tests>.  But you'd have to modify the build being tested in order to integrate this server, which seems suboptimal.

A better option would be to use Python's SimpleHTTPServer module <http://docs.python.org/lib/module-SimpleHTTPServer.html> (or the BaseHTTPServer module, if SimpleHTTPServer is too simple) to implement the server.

SimpleHTTPServer looks pretty easy to use based on this example <http://effbot.org/librarybook/simplehttpserver.htm>.
FWIW, ideally I'd like to be able to run talos directly from my source directory and have it figure out the build to test by parsing my mozconfig file for the objdir without having to copy the talos directory to a new location or modify/create any configuration files.
(In reply to comment #1)
> But you'd have to modify the build being tested in order to integrate this
> server, which seems suboptimal.

You could do as Mochitest does and run it from a separate process via xpcshell if you wanted; see server.js in the mochitest code.  There's really not all that much to it, but if you're in Python already that could be easiest.
Depends on: 396789
Here's the current standalone talos doc:

http://wiki.mozilla.org/StandaloneTalos

It covers two methods of running the tests:
1.  using apache and a local web page set collected/cleaned with wget/bash
2.  using a local proxy server and a proxy cache as a web page test set

This is still quite a ways from not having to edit any config files or have everything run automagically.  I'm hoping that it will become more refined as more people try and run their own perf tests.
Status: NEW → ASSIGNED
Assignee: nobody → anodelman
Status: ASSIGNED → NEW
I've tried running Talos standalone on both Mac and Linux per the doc referenced in comment 4, and it looks like it gets through the Ts and Tp tests, but then it dies with an error about output/out_ts.csv not existing:

h-117:talos myk$ /opt/local/bin/python2.4 run_tests.py standalone/wget_run.config 
Running test: ts
        Screen width/height:1440/900
        colorDepth:24
        Browser inner width/height: 1024/641
        Browser outer width/height: 1024/768
/Users/myk/Mozilla/builds/minefield-opt/dist/Minefield.app/Contents/MacOS/run-mozilla.sh: line 131: 16150 Terminated              "$prog" ${1+"$@"}
Completed test: ts
Running test: tp
        Screen width/height:1440/900
        colorDepth:24
        Browser inner width/height: 1024/641
        Browser outer width/height: 1024/768
Sun Nov  4 20:45:08 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:08 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:08 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:08 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:41 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:41 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:42 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:45:42 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:47:46 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:47:46 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:54:48 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:54:48 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:54:54 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:54:54 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:55:18 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:55:18 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:56:03 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:56:03 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:56:04 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
Sun Nov  4 20:56:04 myk firefox-bin[16187] <Error>: CGImageCreate: invalid image size: 0 x 0.
/Users/myk/Mozilla/builds/minefield-opt/dist/Minefield.app/Contents/MacOS/run-mozilla.sh: line 131: 16187 Terminated              "$prog" ${1+"$@"}
Completed test: tp
Traceback (most recent call last):
  File "run_tests.py", line 318, in ?
    test_file(arg)
  File "run_tests.py", line 307, in test_file
    send_to_csv(csv_file, results)
  File "run_tests.py", line 92, in send_to_csv
    writer = csv.writer(open(csv_file + '_' +  res + '.csv', "wb"))
IOError: [Errno 2] No such file or directory: 'output/out_ts.csv'
does the output subdirectory exist (assuming under "talos" itself)?

This should probably be created by talos if it isn't there.
Yeah, the output directory exists at talos/standalone/output.  It's part of the standalone.zip archive that I expand inside the talos directory.
I've updated the files in standalone.zip to point to the correct directory.  That should work for now, but I'll also put together a patch so that the directory is checked for existence before it bothers running all the tests.
This patch adds checks to ensure that if a csv path is provided it exists, and if a graph server link is provided it is live.  This should stop the sad situation of running the tests to completion only to discover that the results are lost due to a bad path.
Attachment #287607 - Flags: review?(rcampbell)
Comment on attachment 287607 [details] [diff] [review]
check for csv path exists/graph server link existance

one unrelated recommendation in the send_to_csv function, it might be a good idea to split out the different formats into their own functions.

I'd also like to see  command-line switches to send results to the graph-server or to csv.
Attachment #287607 - Flags: review?(rcampbell) → review+
It would be useful to have a "noisy" option that makes Talos print the browser console output to the console instead of swallowing it all.
Checking in post_file.py;
/cvsroot/mozilla/testing/performance/talos/post_file.py,v  <--  post_file.py
new revision: 1.3; previous revision: 1.2
done
Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.17; previous revision: 1.16
done
Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.10; previous revision: 1.9
done
I figured that this was better as a separate switch than --debug - one for talos messages and one for browser messages.  This simply outputs any messages generated by the browser during a given test.

We should consider adding this to the talos command line for the minis, we may be missing info that is being generated during tests that could help us track errors.
Attachment #287926 - Flags: review?(rcampbell)
Comment on attachment 287926 [details] [diff] [review]
adds a --noisy option to output browser messages

I'd use True and False of the contents of NOISY (and DEBUG) for easier, more legible comparisons. Not a show-stopper though.
Attachment #287926 - Flags: review?(rcampbell) → review+
Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.18; previous revision: 1.17
done
Checking in ttest.py;
/cvsroot/mozilla/testing/performance/talos/ttest.py,v  <--  ttest.py
new revision: 1.6; previous revision: 1.5
done
Checking in utils.py;
/cvsroot/mozilla/testing/performance/talos/utils.py,v  <--  utils.py
new revision: 1.3; previous revision: 1.2
done

Checked in as is, will do a utils.py overhaul in the future.
There is a newer, simpler version of standalone talos up at:

http://wiki.mozilla.org/StandaloneTalos

No longer has any dependencies on web servers.  I've also tried to contain everything into a single zip file instead of having users check out their own copies of the build.
Priority: -- → P3
Updated standalone talos to v 1.1 - includes support for new file:// security restrictions along with an update of the process management code within talos itself.

Again, see:

http://wiki.mozilla.org/StandaloneTalos
Assignee: anodelman → nobody
Component: Testing → Release Engineering: Talos
Product: Core → mozilla.org
QA Contact: testing → release
Version: Trunk → other
Assignee: nobody → anodelman
The general standalone talos framework is complete.  Updates to standalone talos can be made in new bugs.  
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Component: Release Engineering: Talos → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: