Closed Bug 368720 Opened 14 years ago Closed 13 years ago
python test framework integration with graph server/new web page set
This patch covers the simplifications that I've made to the python performance testing framework to make use of the graph server and new web page test set. I've pulled out all the local code that was handling graph creation and just sending the collected data to a graph server whose link is specified in paths.py. It also adds some configuration options to: - add the branch and buildid to each test run - allows for a different profile to be used for each test run
Added support for a schema change that I made to the graph server - needed to handle the 'type' field. Also, fixed a problem with the list of webpages to be cycled through.
This is the code currently being used to run the performance tests in buildbot. It provides a framework for working with new pages sets along with integration with the new graph server.
This patch adds support for Linux to the testing framework. The OS is specified in paths.py. Alice: I tried posting results to the graph server you showed me a few months ago. The results never showed up on the web interface. None of the graph server code has been touched by my patch so I'm assuming something on the graph server end has changed. However, I thought I should make you aware of it.
I should also note that the above patch needs to be applied after applying Alice's latest patch.
I added comments to the README and paths.py describing the 'OS' variable in paths.py
My previous patch was not created correctly (I'd created it against a different directory structure, which then caused it to be difficult to apply to the existing code in CVS). So, no code changes here - just diff silliness.
So, funny story, I created the old patch against a not-updated win32 directory. So, now this should be the latest code in a diff that was created against 1) the latest cvs code and 2) the correct directory structure. Very sorry for the bugmail spam about this.
Updated against alice's latest diff.
Sorry for the bugspam everyone. This patch fixes a bug where the Tp tests would run but no performance information would be collected.
This patch renames paths.py to config.py to centralize all normal configuration. Comments have been added to that file regarding counter types for Linux.
This patch covers both my changes to the python framework and the port made by bhearsum. I've tested it on both windows and linux and it ran successfully.
I need to go over it more carefully, but the code itself looks good :) My only concerns right now are structural nit-pick kind of stuff, that I think will make the project more approachable: I find it confusing having both linux and win32 specific code in the win32 dir. It's probably be worth rethinking the directory structure.. I'd create a new directory (and give this project a name), and has the nice side-effect of leaving the win32 dir alone for now (we can remove it once we've confirmed that noone is using it). I can do this for you if you make up a good name :) Basically I would just import what's currently in "mozilla/testing/performance/win32" and then apply your patch, so the history is more useful. Having the "#!c:\Python24\python.exe" is kind of annoying as it only works in Windows if you have python installed there. Might as well leave that out, and require it be started with "python run_test.py". This is certainly not a blocker to checking in, but it'd sure be nice to not have to configure so much on non-win32.. the OS could be detected for example, and there could be default per-OS paths. The defaults look reasonable for win32 though so that's a good start.
I like your idea of shifting it into a directory with a new project name. I'm not very good with project names, but in QA we seem to be in something of a greek myth kick so I would suggest 'Talos' after the bronze automaton that hurled rocks at unwary seamen. On the #!c:\Python24\python.exe issue, I haven't had any trouble running it under linux with 'python run_tests.py config.yml' - so I don't think that it actually requires any extra configuration, maybe just a note in the README.
(In reply to comment #13) > I like your idea of shifting it into a directory with a new project name. I'm > not very good with project names, but in QA we seem to be in something of a > greek myth kick so I would suggest 'Talos' after the bronze automaton that > hurled rocks at unwary seamen. That sounds ideal. > On the #!c:\Python24\python.exe issue, I haven't had any trouble running it > under linux with 'python run_tests.py config.yml' - so I don't think that it > actually requires any extra configuration, maybe just a note in the README. I was thinking on win32 if you don't have python installed into c:\python24, it's easier to just have the instructions for all OSes be "python whatever.py" instead of having to bump that path for different versions of python :)
Removed "#!c:/Python24/python.exe" from various files. Updated README to instruct user to run with 'python run_tests.py'. When checking this stuff in be sure to remove the page_load_test/base directory - when I've applied the patch to my own stuff I don't seem to get that directory to get deleted, I think that it might have some binary files hanging around in it that patch doesn't know what to do with.
Attachment #263781 - Flags: review?(rhelmer) → review+
I've created and imported a snapshot of mozilla/testing/performance/win32 into mozilla/testing/performance/talos, just confirmed that everything is the same except I've skipped the page_load_test/base directory.
Here's the patch I'm going to land. Should be functionally equivalent to attachment 263781 [details] [diff] [review] except: * top-level dir is win32->talos * no more page_load_test/base Plus I used CVS to log the adds/removes, which uses the "/dev/null" trick.
Attachment #263781 - Attachment is obsolete: true
Landed: Checking in README.txt; /cvsroot/mozilla/testing/performance/talos/README.txt,v <-- README.txt new revision: 1.2; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/config.py,v done Checking in config.py; /cvsroot/mozilla/testing/performance/talos/config.py,v <-- config.py initial revision: 1.1 done Checking in ffprocess.py; /cvsroot/mozilla/testing/performance/talos/ffprocess.py,v <-- ffprocess.py new revision: 1.2; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v done Checking in ffprocess_linux.py; /cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v <-- ffprocess_linux.py initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v done Checking in ffprocess_win32.py; /cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v <-- ffprocess_win32.py initial revision: 1.1 done Checking in ffprofile.py; /cvsroot/mozilla/testing/performance/talos/ffprofile.py,v <-- ffprofile.py new revision: 1.2; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v done Checking in ffprofile_linux.py; /cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v <-- ffprofile_linux.py initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v done Checking in ffprofile_win32.py; /cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v <-- ffprofile_win32.py initial revision: 1.1 done Checking in initialize.html; /cvsroot/mozilla/testing/performance/talos/initialize.html,v <-- initialize.html new revision: 1.2; previous revision: 1.1 done Removing paths.py; /cvsroot/mozilla/testing/performance/talos/paths.py,v <-- paths.py new revision: delete; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/post_file.py,v done Checking in post_file.py; /cvsroot/mozilla/testing/performance/talos/post_file.py,v <-- post_file.py initial revision: 1.1 done Removing report.py; /cvsroot/mozilla/testing/performance/talos/report.py,v <-- report.py new revision: delete; previous revision: 1.1 done Checking in run_tests.py; /cvsroot/mozilla/testing/performance/talos/run_tests.py,v <-- run_tests.py new revision: 1.2; previous revision: 1.1 done Checking in sample.config; /cvsroot/mozilla/testing/performance/talos/sample.config,v <-- sample.config new revision: 1.2; previous revision: 1.1 done Checking in tp.py; /cvsroot/mozilla/testing/performance/talos/tp.py,v <-- tp.py new revision: 1.2; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/tp_linux.py,v done Checking in tp_linux.py; /cvsroot/mozilla/testing/performance/talos/tp_linux.py,v <-- tp_linux.py initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/tp_win32.py,v done Checking in tp_win32.py; /cvsroot/mozilla/testing/performance/talos/tp_win32.py,v <-- tp_win32.py initial revision: 1.1 done Checking in ts.py; /cvsroot/mozilla/testing/performance/talos/ts.py,v <-- ts.py new revision: 1.2; previous revision: 1.1 done Removing page_load_test/cycler.html; /cvsroot/mozilla/testing/performance/talos/page_load_test/cycler.html,v <-- cycler.html new revision: delete; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v done Checking in page_load_test/framecycler.html; /cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v <-- framecycler.html initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v done Checking in page_load_test/parray.js; /cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v <-- parray.js initial revision: 1.1 done Checking in page_load_test/report.html; /cvsroot/mozilla/testing/performance/talos/page_load_test/report.html,v <-- report.html new revision: 1.2; previous revision: 1.1 done Checking in startup_test/startup_test.html; /cvsroot/mozilla/testing/performance/talos/startup_test/startup_test.html,v <-- startup_test.html new revision: 1.2; previous revision: 1.1 done
Added some links and a little description of the licensing that covers post_file.py.
Attachment #263792 - Flags: review?(rhelmer)
Attachment #263792 - Flags: review?(rhelmer) → review+
(In reply to comment #19) > Created an attachment (id=263792) [details] > post_file.py license clarification > > Added some links and a little description of the licensing that covers > post_file.py. > Checked in.
Is this bug meant to be resolved fixed?
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Assignee: nobody → anodelman
Component: Testing → Release Engineering: Talos
Product: Core → mozilla.org
QA Contact: testing → release
Version: Trunk → other
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.