Closed Bug 368720 Opened 14 years ago Closed 13 years ago

python test framework integration with graph server/new web page set

Categories

(Release Engineering :: General, defect)

x86
All
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: anodelman, Assigned: anodelman)

References

Details

Attachments

(2 files, 12 obsolete files)

130.04 KB, patch
Details | Diff | Splinter Review
643 bytes, patch
rhelmer
: review+
Details | Diff | Splinter Review
This patch covers the simplifications that I've made to the python performance testing framework to make use of the graph server and new web page test set.  I've pulled out all the local code that was handling graph creation and just sending the collected data to a graph server whose link is specified in paths.py.

It also adds some configuration options to:
- add the branch and buildid to each test run
- allows for a different profile to be used for each test run
Attachment #253363 - Flags: review?(annie.sullivan)
Added support for a schema change that I made to the graph server - needed to handle the 'type' field.

Also, fixed a problem with the list of webpages to be cycled through.
Attachment #253363 - Attachment is obsolete: true
Attachment #253705 - Flags: review?(annie.sullivan)
Attachment #253363 - Flags: review?(annie.sullivan)
This is the code currently being used to run the performance tests in buildbot.  It provides a framework for working with new pages sets along with integration with the new graph server.
Attachment #253705 - Attachment is obsolete: true
Attachment #262044 - Flags: review?(annie.sullivan)
Attachment #253705 - Flags: review?(annie.sullivan)
Attached patch adds support for linux (obsolete) — Splinter Review
This patch adds support for Linux to the testing framework. The OS is specified in paths.py.

Alice: I tried posting results to the graph server you showed me a few months ago. The results never showed up on the web interface. None of the graph server code has been touched by my patch so I'm assuming something on the graph server end has changed. However, I thought I should make you aware of it.
Attachment #262202 - Flags: review?(anodelman)
I should also note that the above patch needs to be applied after applying Alice's latest patch.
Attached patch same patch, updated README (obsolete) — Splinter Review
I added comments to the README and paths.py describing the 'OS' variable in paths.py
Attachment #262202 - Attachment is obsolete: true
Attachment #262294 - Flags: review?(anodelman)
Attachment #262202 - Flags: review?(anodelman)
My previous patch was not created correctly (I'd created it against a different directory structure, which then caused it to be difficult to apply to the existing  code in CVS).

So, no code changes here - just diff silliness.
Attachment #262044 - Attachment is obsolete: true
Attachment #262508 - Flags: review?(annie.sullivan)
Attachment #262044 - Flags: review?(annie.sullivan)
So, funny story, I created the old patch against a not-updated win32 directory.  So, now this should be the latest code in a diff that was created against 1) the latest cvs code and 2) the correct directory structure.

Very sorry for the bugmail spam about this.
Attachment #262508 - Attachment is obsolete: true
Attachment #262513 - Flags: review?(annie.sullivan)
Attachment #262508 - Flags: review?(annie.sullivan)
Updated against alice's latest diff.
Attachment #262294 - Attachment is obsolete: true
Attachment #262521 - Flags: review?(anodelman)
Attachment #262294 - Flags: review?(anodelman)
Sorry for the bugspam everyone. This patch fixes a bug where the Tp tests would run but no performance information would be collected.
Attachment #262521 - Attachment is obsolete: true
Attachment #262531 - Flags: review?(anodelman)
Attachment #262521 - Flags: review?(anodelman)
This patch renames paths.py to config.py to centralize all normal configuration. Comments have been added to that file regarding counter types for Linux.
Attachment #262531 - Attachment is obsolete: true
Attachment #262813 - Flags: review?(anodelman)
Attachment #262531 - Flags: review?(anodelman)
This patch covers both my changes to the python framework and the port made by bhearsum.  I've tested it on both windows and linux and it ran successfully.
Attachment #262513 - Attachment is obsolete: true
Attachment #262813 - Attachment is obsolete: true
Attachment #263680 - Flags: review?(rhelmer)
Attachment #262513 - Flags: review?(annie.sullivan)
Attachment #262813 - Flags: review?(anodelman)
I need to go over it more carefully, but the code itself looks good :) My only concerns right now are structural nit-pick kind of stuff, that I think will make the project more approachable:

I find it confusing having both linux and win32 specific code in the win32 dir. It's probably be worth rethinking the directory structure.. I'd create a new directory (and give this project a name), and has the nice side-effect of leaving the win32 dir alone for now (we can remove it once we've confirmed that noone is using it). 

I can do this for you if you make up a good name :) Basically I would just import what's currently in "mozilla/testing/performance/win32" and then apply your patch, so the history is more useful.

Having the "#!c:\Python24\python.exe" is kind of annoying as it only works in Windows if you have python installed there. Might as well leave that out, and require it be started with "python run_test.py".

This is certainly not a blocker to checking in, but it'd sure be nice to not have to configure so much on non-win32.. the OS could be detected for example, and there could be default per-OS paths. The defaults look reasonable for win32 though so that's a good start.
I like your idea of shifting it into a directory with a new project name.  I'm not very good with project names, but in QA we seem to be in something of a greek myth kick so I would suggest 'Talos' after the bronze automaton that hurled rocks at unwary seamen.

On the #!c:\Python24\python.exe issue, I haven't had any trouble running it under linux with 'python run_tests.py config.yml' - so I don't think that it actually requires any extra configuration, maybe just a note in the README.
(In reply to comment #13)
> I like your idea of shifting it into a directory with a new project name.  I'm
> not very good with project names, but in QA we seem to be in something of a
> greek myth kick so I would suggest 'Talos' after the bronze automaton that
> hurled rocks at unwary seamen.

That sounds ideal.

> On the #!c:\Python24\python.exe issue, I haven't had any trouble running it
> under linux with 'python run_tests.py config.yml' - so I don't think that it
> actually requires any extra configuration, maybe just a note in the README.

I was thinking on win32 if you don't have python installed into c:\python24, it's easier to just have the instructions for all OSes be "python whatever.py" instead of having to bump that path for different versions of python :) 
Removed "#!c:/Python24/python.exe" from various files.  Updated README to instruct user to run with 'python run_tests.py'.

When checking this stuff in be sure to remove the page_load_test/base directory - when I've applied the patch to my own stuff I don't seem to get that directory to get deleted, I think that it might have some binary files hanging around in it that patch doesn't know what to do with.
Attachment #263680 - Attachment is obsolete: true
Attachment #263781 - Flags: review?(rhelmer)
Attachment #263680 - Flags: review?(rhelmer)
Attachment #263781 - Flags: review?(rhelmer) → review+
I've created and imported a snapshot of mozilla/testing/performance/win32 into mozilla/testing/performance/talos, just confirmed that everything is the same except I've skipped the page_load_test/base directory.
Here's the patch I'm going to land. Should be functionally equivalent to attachment 263781 [details] [diff] [review] except:

* top-level dir is win32->talos
* no more page_load_test/base

Plus I used CVS to log the adds/removes, which uses the "/dev/null" trick.
Attachment #263781 - Attachment is obsolete: true
Landed:

Checking in README.txt;
/cvsroot/mozilla/testing/performance/talos/README.txt,v  <--  README.txt
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/config.py,v
done
Checking in config.py;
/cvsroot/mozilla/testing/performance/talos/config.py,v  <--  config.py
initial revision: 1.1
done
Checking in ffprocess.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess.py,v  <--  ffprocess.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v
done
Checking in ffprocess_linux.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v  <--  ffprocess_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v
done
Checking in ffprocess_win32.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v  <--  ffprocess_win32.py
initial revision: 1.1
done
Checking in ffprofile.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile.py,v  <--  ffprofile.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v
done
Checking in ffprofile_linux.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v  <--  ffprofile_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v
done
Checking in ffprofile_win32.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v  <--  ffprofile_win32.py
initial revision: 1.1
done
Checking in initialize.html;
/cvsroot/mozilla/testing/performance/talos/initialize.html,v  <--  initialize.html
new revision: 1.2; previous revision: 1.1
done
Removing paths.py;
/cvsroot/mozilla/testing/performance/talos/paths.py,v  <--  paths.py
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/post_file.py,v
done
Checking in post_file.py;
/cvsroot/mozilla/testing/performance/talos/post_file.py,v  <--  post_file.py
initial revision: 1.1
done
Removing report.py;
/cvsroot/mozilla/testing/performance/talos/report.py,v  <--  report.py
new revision: delete; previous revision: 1.1
done
Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.2; previous revision: 1.1
done
Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.2; previous revision: 1.1
done
Checking in tp.py;
/cvsroot/mozilla/testing/performance/talos/tp.py,v  <--  tp.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/tp_linux.py,v
done
Checking in tp_linux.py;
/cvsroot/mozilla/testing/performance/talos/tp_linux.py,v  <--  tp_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/tp_win32.py,v
done
Checking in tp_win32.py;
/cvsroot/mozilla/testing/performance/talos/tp_win32.py,v  <--  tp_win32.py
initial revision: 1.1
done
Checking in ts.py;
/cvsroot/mozilla/testing/performance/talos/ts.py,v  <--  ts.py
new revision: 1.2; previous revision: 1.1
done
Removing page_load_test/cycler.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/cycler.html,v  <--  cycler.html
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v
done
Checking in page_load_test/framecycler.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v  <--  framecycler.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v
done
Checking in page_load_test/parray.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v  <--  parray.js
initial revision: 1.1
done
Checking in page_load_test/report.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/report.html,v  <--  report.html
new revision: 1.2; previous revision: 1.1
done
Checking in startup_test/startup_test.html;
/cvsroot/mozilla/testing/performance/talos/startup_test/startup_test.html,v  <--  startup_test.html
new revision: 1.2; previous revision: 1.1
done
Added some links and a little description of the licensing that covers post_file.py.
Attachment #263792 - Flags: review?(rhelmer)
Attachment #263792 - Flags: review?(rhelmer) → review+
(In reply to comment #19)
> Created an attachment (id=263792) [details]
> post_file.py license clarification
> 
> Added some links and a little description of the licensing that covers
> post_file.py.
> 

Checked in.
Is this bug meant to be resolved fixed?
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Assignee: nobody → anodelman
Component: Testing → Release Engineering: Talos
Product: Core → mozilla.org
QA Contact: testing → release
Version: Trunk → other
Component: Release Engineering: Talos → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.