Last Comment Bug 493057 - Run fennecmark regularly
: Run fennecmark regularly
Status: VERIFIED FIXED
:
Product: Fennec Graveyard
Classification: Graveyard
Component: General (show other bugs)
: Trunk
: x86 Linux
: -- normal (vote)
: ---
Assigned To: Joel Maher ( :jmaher)
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-14 12:48 PDT by (dormant account)
Modified: 2009-08-21 12:21 PDT (History)
5 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
FennecMark 1.0 (12.39 KB, patch)
2009-06-04 20:25 PDT, Joel Maher ( :jmaher)
no flags Details | Diff | Splinter Review
fennecbench.patch 1.0 (25.21 KB, patch)
2009-06-15 13:36 PDT, Joel Maher ( :jmaher)
no flags Details | Diff | Splinter Review
fennecbench.manifest 1.0 (507 bytes, patch)
2009-06-19 14:31 PDT, Joel Maher ( :jmaher)
no flags Details | Diff | Splinter Review

Description (dormant account) 2009-05-14 12:48:33 PDT

    
Comment 1 (dormant account) 2009-05-14 13:01:30 PDT
Here is a blog post describing it http://blog.mozilla.com/tglek/2009/04/23/benchmarks-and-instrumentation-for-fennecetc/

Basically it is a benchmark that measures and outputs interesting performance numbers, would like to get those numbers graphed for easy comparison.
Comment 2 Joel Maher ( :jmaher) 2009-06-04 20:25:52 PDT
Created attachment 381689 [details] [diff] [review]
FennecMark 1.0

This is the first round of fennecmark 1.0.  This reports two numbers:
1) median pan lag time
2) median zoom lag time

There is an option to report a third number: total pan time.

This patch needs to be applied at the root of this repository:
http://hg.mozilla.org/users/asasaki_mozilla.com/talos-maemo/

Developed and tested on Ubuntu Linux.  For the .jar file, this makes a clean install, but we need to resolve the core source code location.
Comment 3 Aki Sasaki [:aki] 2009-06-08 21:28:02 PDT
Joel: have you run these at all on an n810?
I'm hitting timeouts for both tests.  Bumped the timeout to 300, still timing out; trying 1200 now.
Comment 4 Aki Sasaki [:aki] 2009-06-09 15:18:29 PDT
Ok, now I'm getting a 404 for http://localhost/page_load_test/pages/en.wikipedia.org_wiki_Main_Page/en.wikipedia.org/wiki/Main_Page.html

This page doesn't exist in the talos pageset on the devices. Did you get this from somewhere or is it in the standalone pageset?

Is there something special about this page or could I choose any page off the existing pageset?
Comment 5 Joel Maher ( :jmaher) 2009-06-09 18:16:28 PDT
updating the loadURI to point to a page on your device works just fine!  The one problem I do have (while running with -d -n) could be this:

EBUG: Working with test: tzoom
DEBUG: Sending results: 855
Generating results file: tzoom: 
		Started Wed, 10 Jun 2009 17:52:42
Traceback (most recent call last):
  File "run_tests.py", line 541, in <module>
    test_file(arg)
  File "run_tests.py", line 520, in test_file
    links = send_to_graph(results_server, results_link, title, date, browser_config, results)
  File "run_tests.py", line 229, in send_to_graph
    files.append(construct_file(machine, fullname, browser_config['branch_name'], browser_config['sourcestamp'], browser_config['buildid'], date, vals))
KeyError: 'branch_name'
maemo-n810-07:/media/mmc1/talos#
Comment 6 Aki Sasaki [:aki] 2009-06-10 14:34:22 PDT
Ok, I'm not sure if you got this via IRC; putting here for clarity.

[10:54]	<aki>	i don't think i set branchName
[10:56]	oh, apparently PerfConfigurator.py -b sets both branch and branch_name
[10:58]	jmaher: i run |python PerfConfigurator.py -v -e /media/mmc1/fennec/fennec -t `hostname` -b mobile-browser --activeTests tzoom --sampleConfig mobile.config --browserWait 60 --noChrome --oldresultsServer graphs-stage.mozilla.org --oldresultsLink /server/bulk.cgi --output local.config|
[11:02]	[ERROR]	Connection to ircs://irc.mozilla.org:6697/ (ircs://irc.mozilla.org:6697/) timed out. [Help]
Comment 7 Joel Maher ( :jmaher) 2009-06-10 14:43:34 PDT
I will poke around the PerfConfigurator a bit to see if there is something we can do.  Is there a chance that the server needs to be configured for the new test name?
Comment 8 Aki Sasaki [:aki] 2009-06-10 15:24:14 PDT
I *think* the current graph server will take new tests without any manual intervention. Alice?

We are moving to a new graph server for Firefox talos.  There's a whole bunch of stuff we'll have to do before we can report there from the mobile devices... maybe Q3, maybe not.  To be tracked in bug 487335.
Comment 9 Joel Maher ( :jmaher) 2009-06-12 10:59:30 PDT
in #perfomatic, alice helped me debug the issue.  bug 497922 address the issue required to fix this.

Basically in run_tests.py there is a hardcoded list of tests 'ts' and 'twinopen' that are used to match tests that report in a certain method.

modifying this list of tests to include tzoom and tpan appear to work:
http://graphs-stage.mozilla.org/graph.html#show=833011 <- tzoom.

I also increased the cycles from 1 to 5 in order to get that result.  I am going to recommend for 10 cycles.

Unfortunately this requires another file change which will need to get rolled out to all the talos devices.
Comment 10 Joel Maher ( :jmaher) 2009-06-15 13:36:39 PDT
Created attachment 383337 [details] [diff] [review]
fennecbench.patch 1.0

this patch applies to the http://hg.mozilla.org/users/asasaki_mozilla.com/talos-maemo/ tree.  if you run ./build.sh inside of tpan/fennecmark, you will get a .jar file.  Copy the fennecmark.jar and fennecmark.manifest to the tpan/chrome folder.


Also I have fixed the noisy output of tpan.js. I suspect this might fix the results with tpan
Comment 11 Aki Sasaki [:aki] 2009-06-15 14:40:03 PDT
This is checked in and I've pushed the new jar to the n810s.
When running build.sh, I don't get a fennecmark.manifest, however.
Comment 12 Joel Maher ( :jmaher) 2009-06-19 14:31:06 PDT
Created attachment 384163 [details] [diff] [review]
fennecbench.manifest 1.0

This adds fennecbench.manifest to the tpan/fennecmark directory.  This will be inline with the instructions that build.sh display.
Comment 13 Aki Sasaki [:aki] 2009-06-22 14:55:00 PDT
That's checked in, and tests are running.
Comment 14 Joel Maher ( :jmaher) 2009-08-21 12:21:27 PDT
verfied with beta3.  There is a bug tracking the issues with the latest updates.

Note You need to log in before you can comment on or make changes to this bug.