368720 - python test framework integration with graph server/new web page set

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[alice nodelman [:alice] [:anode]]

Attachment: python test framework graph server/page set integration

This patch covers the simplifications that I've made to the python performance testing framework to make use of the graph server and new web page test set.  I've pulled out all the local code that was handling graph creation and just sending the collected data to a graph server whose link is specified in paths.py.

It also adds some configuration options to:
- add the branch and buildid to each test run
- allows for a different profile to be used for each test run

Comment 1

[alice nodelman [:alice] [:anode]]

Attachment: attempt #2 :python test framework graph server/page set integration

Added support for a schema change that I made to the graph server - needed to handle the 'type' field.

Also, fixed a problem with the list of webpages to be cycled through.

Comment 2

[alice nodelman [:alice] [:anode]]

Attachment: attempt #3 :python test framework graph server/page set integration

This is the code currently being used to run the performance tests in buildbot.  It provides a framework for working with new pages sets along with integration with the new graph server.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: adds support for linux

This patch adds support for Linux to the testing framework. The OS is specified in paths.py.

Alice: I tried posting results to the graph server you showed me a few months ago. The results never showed up on the web interface. None of the graph server code has been touched by my patch so I'm assuming something on the graph server end has changed. However, I thought I should make you aware of it.

Comment 4

[[e-mail] (:bhearsum)]

I should also note that the above patch needs to be applied after applying Alice's latest patch.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: same patch, updated README

I added comments to the README and paths.py describing the 'OS' variable in paths.py

Comment 6

[alice nodelman [:alice] [:anode]]

Attachment: attempt #4 :python test framework graph server/page set integration

My previous patch was not created correctly (I'd created it against a different directory structure, which then caused it to be difficult to apply to the existing  code in CVS).

So, no code changes here - just diff silliness.

Comment 7

[alice nodelman [:alice] [:anode]]

Attachment: attempt #5 :python test framework graph server/page set integration

So, funny story, I created the old patch against a not-updated win32 directory.  So, now this should be the latest code in a diff that was created against 1) the latest cvs code and 2) the correct directory structure.

Very sorry for the bugmail spam about this.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: updated against alice's latest diff

Updated against alice's latest diff.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: fix bug where Tp performance information was not collected

Sorry for the bugspam everyone. This patch fixes a bug where the Tp tests would run but no performance information would be collected.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: renamed paths.py to config.py, added comments

This patch renames paths.py to config.py to centralize all normal configuration. Comments have been added to that file regarding counter types for Linux.

Comment 11

[alice nodelman [:alice] [:anode]]

Attachment: attempt #6 :python test framework graph server/page set integration + linux port

This patch covers both my changes to the python framework and the port made by bhearsum.  I've tested it on both windows and linux and it ran successfully.

Comment 12

[Robert Helmer [:rhelmer]]

I need to go over it more carefully, but the code itself looks good :) My only concerns right now are structural nit-pick kind of stuff, that I think will make the project more approachable:

I find it confusing having both linux and win32 specific code in the win32 dir. It's probably be worth rethinking the directory structure.. I'd create a new directory (and give this project a name), and has the nice side-effect of leaving the win32 dir alone for now (we can remove it once we've confirmed that noone is using it). 

I can do this for you if you make up a good name :) Basically I would just import what's currently in "mozilla/testing/performance/win32" and then apply your patch, so the history is more useful.

Having the "#!c:\Python24\python.exe" is kind of annoying as it only works in Windows if you have python installed there. Might as well leave that out, and require it be started with "python run_test.py".

This is certainly not a blocker to checking in, but it'd sure be nice to not have to configure so much on non-win32.. the OS could be detected for example, and there could be default per-OS paths. The defaults look reasonable for win32 though so that's a good start.

Comment 13

[alice nodelman [:alice] [:anode]]

I like your idea of shifting it into a directory with a new project name.  I'm not very good with project names, but in QA we seem to be in something of a greek myth kick so I would suggest 'Talos' after the bronze automaton that hurled rocks at unwary seamen.

On the #!c:\Python24\python.exe issue, I haven't had any trouble running it under linux with 'python run_tests.py config.yml' - so I don't think that it actually requires any extra configuration, maybe just a note in the README.

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to comment #13)
> I like your idea of shifting it into a directory with a new project name.  I'm
> not very good with project names, but in QA we seem to be in something of a
> greek myth kick so I would suggest 'Talos' after the bronze automaton that
> hurled rocks at unwary seamen.

That sounds ideal.

> On the #!c:\Python24\python.exe issue, I haven't had any trouble running it
> under linux with 'python run_tests.py config.yml' - so I don't think that it
> actually requires any extra configuration, maybe just a note in the README.

I was thinking on win32 if you don't have python installed into c:\python24, it's easier to just have the instructions for all OSes be "python whatever.py" instead of having to bump that path for different versions of python :) 

Comment 15

[alice nodelman [:alice] [:anode]]

Attachment: attempt #7 :python test framework graph server/page set integration + linux port

Removed "#!c:/Python24/python.exe" from various files.  Updated README to instruct user to run with 'python run_tests.py'.

When checking this stuff in be sure to remove the page_load_test/base directory - when I've applied the patch to my own stuff I don't seem to get that directory to get deleted, I think that it might have some binary files hanging around in it that patch doesn't know what to do with.

Comment 16

[Robert Helmer [:rhelmer]]

I've created and imported a snapshot of mozilla/testing/performance/win32 into mozilla/testing/performance/talos, just confirmed that everything is the same except I've skipped the page_load_test/base directory.

Comment 17

[Robert Helmer [:rhelmer]]

Attachment: make patch apply to "talos" directory

Here's the patch I'm going to land. Should be functionally equivalent to attachment 263781 [details] [diff] [review] except:

* top-level dir is win32->talos
* no more page_load_test/base

Plus I used CVS to log the adds/removes, which uses the "/dev/null" trick.

Comment 18

[Robert Helmer [:rhelmer]]

Landed:

Checking in README.txt;
/cvsroot/mozilla/testing/performance/talos/README.txt,v  <--  README.txt
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/config.py,v
done
Checking in config.py;
/cvsroot/mozilla/testing/performance/talos/config.py,v  <--  config.py
initial revision: 1.1
done
Checking in ffprocess.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess.py,v  <--  ffprocess.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v
done
Checking in ffprocess_linux.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess_linux.py,v  <--  ffprocess_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v
done
Checking in ffprocess_win32.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess_win32.py,v  <--  ffprocess_win32.py
initial revision: 1.1
done
Checking in ffprofile.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile.py,v  <--  ffprofile.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v
done
Checking in ffprofile_linux.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile_linux.py,v  <--  ffprofile_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v
done
Checking in ffprofile_win32.py;
/cvsroot/mozilla/testing/performance/talos/ffprofile_win32.py,v  <--  ffprofile_win32.py
initial revision: 1.1
done
Checking in initialize.html;
/cvsroot/mozilla/testing/performance/talos/initialize.html,v  <--  initialize.html
new revision: 1.2; previous revision: 1.1
done
Removing paths.py;
/cvsroot/mozilla/testing/performance/talos/paths.py,v  <--  paths.py
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/post_file.py,v
done
Checking in post_file.py;
/cvsroot/mozilla/testing/performance/talos/post_file.py,v  <--  post_file.py
initial revision: 1.1
done
Removing report.py;
/cvsroot/mozilla/testing/performance/talos/report.py,v  <--  report.py
new revision: delete; previous revision: 1.1
done
Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.2; previous revision: 1.1
done
Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.2; previous revision: 1.1
done
Checking in tp.py;
/cvsroot/mozilla/testing/performance/talos/tp.py,v  <--  tp.py
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/tp_linux.py,v
done
Checking in tp_linux.py;
/cvsroot/mozilla/testing/performance/talos/tp_linux.py,v  <--  tp_linux.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/tp_win32.py,v
done
Checking in tp_win32.py;
/cvsroot/mozilla/testing/performance/talos/tp_win32.py,v  <--  tp_win32.py
initial revision: 1.1
done
Checking in ts.py;
/cvsroot/mozilla/testing/performance/talos/ts.py,v  <--  ts.py
new revision: 1.2; previous revision: 1.1
done
Removing page_load_test/cycler.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/cycler.html,v  <--  cycler.html
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v
done
Checking in page_load_test/framecycler.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/framecycler.html,v  <--  framecycler.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v
done
Checking in page_load_test/parray.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/parray.js,v  <--  parray.js
initial revision: 1.1
done
Checking in page_load_test/report.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/report.html,v  <--  report.html
new revision: 1.2; previous revision: 1.1
done
Checking in startup_test/startup_test.html;
/cvsroot/mozilla/testing/performance/talos/startup_test/startup_test.html,v  <--  startup_test.html
new revision: 1.2; previous revision: 1.1
done

Comment 19

[alice nodelman [:alice] [:anode]]

Attachment: post_file.py license clarification

Added some links and a little description of the licensing that covers post_file.py.

Comment 20

[Robert Helmer [:rhelmer]]

(In reply to comment #19)
> Created an attachment (id=263792) [details]
> post_file.py license clarification
> 
> Added some links and a little description of the licensing that covers
> post_file.py.
> 

Checked in.

Comment 21

[Mike Schroepfer]

Is this bug meant to be resolved fixed?