Here is a blog post describing it http://blog.mozilla.com/tglek/2009/04/23/benchmarks-and-instrumentation-for-fennecetc/ Basically it is a benchmark that measures and outputs interesting performance numbers, would like to get those numbers graphed for easy comparison.
Created attachment 381689 [details] [diff] [review] FennecMark 1.0 This is the first round of fennecmark 1.0. This reports two numbers: 1) median pan lag time 2) median zoom lag time There is an option to report a third number: total pan time. This patch needs to be applied at the root of this repository: http://hg.mozilla.org/users/asasaki_mozilla.com/talos-maemo/ Developed and tested on Ubuntu Linux. For the .jar file, this makes a clean install, but we need to resolve the core source code location.
Joel: have you run these at all on an n810? I'm hitting timeouts for both tests. Bumped the timeout to 300, still timing out; trying 1200 now.
Ok, now I'm getting a 404 for http://localhost/page_load_test/pages/en.wikipedia.org_wiki_Main_Page/en.wikipedia.org/wiki/Main_Page.html This page doesn't exist in the talos pageset on the devices. Did you get this from somewhere or is it in the standalone pageset? Is there something special about this page or could I choose any page off the existing pageset?
updating the loadURI to point to a page on your device works just fine! The one problem I do have (while running with -d -n) could be this: EBUG: Working with test: tzoom DEBUG: Sending results: 855 Generating results file: tzoom: Started Wed, 10 Jun 2009 17:52:42 Traceback (most recent call last): File "run_tests.py", line 541, in <module> test_file(arg) File "run_tests.py", line 520, in test_file links = send_to_graph(results_server, results_link, title, date, browser_config, results) File "run_tests.py", line 229, in send_to_graph files.append(construct_file(machine, fullname, browser_config['branch_name'], browser_config['sourcestamp'], browser_config['buildid'], date, vals)) KeyError: 'branch_name' maemo-n810-07:/media/mmc1/talos#
Ok, I'm not sure if you got this via IRC; putting here for clarity. [10:54] <aki> i don't think i set branchName [10:56] oh, apparently PerfConfigurator.py -b sets both branch and branch_name [10:58] jmaher: i run |python PerfConfigurator.py -v -e /media/mmc1/fennec/fennec -t `hostname` -b mobile-browser --activeTests tzoom --sampleConfig mobile.config --browserWait 60 --noChrome --oldresultsServer graphs-stage.mozilla.org --oldresultsLink /server/bulk.cgi --output local.config| [11:02] [ERROR] Connection to ircs://irc.mozilla.org:6697/ (ircs://irc.mozilla.org:6697/) timed out. [Help]
I will poke around the PerfConfigurator a bit to see if there is something we can do. Is there a chance that the server needs to be configured for the new test name?
I *think* the current graph server will take new tests without any manual intervention. Alice? We are moving to a new graph server for Firefox talos. There's a whole bunch of stuff we'll have to do before we can report there from the mobile devices... maybe Q3, maybe not. To be tracked in bug 487335.
in #perfomatic, alice helped me debug the issue. bug 497922 address the issue required to fix this. Basically in run_tests.py there is a hardcoded list of tests 'ts' and 'twinopen' that are used to match tests that report in a certain method. modifying this list of tests to include tzoom and tpan appear to work: http://graphs-stage.mozilla.org/graph.html#show=833011 <- tzoom. I also increased the cycles from 1 to 5 in order to get that result. I am going to recommend for 10 cycles. Unfortunately this requires another file change which will need to get rolled out to all the talos devices.
Created attachment 383337 [details] [diff] [review] fennecbench.patch 1.0 this patch applies to the http://hg.mozilla.org/users/asasaki_mozilla.com/talos-maemo/ tree. if you run ./build.sh inside of tpan/fennecmark, you will get a .jar file. Copy the fennecmark.jar and fennecmark.manifest to the tpan/chrome folder. Also I have fixed the noisy output of tpan.js. I suspect this might fix the results with tpan
This is checked in and I've pushed the new jar to the n810s. When running build.sh, I don't get a fennecmark.manifest, however.
Created attachment 384163 [details] [diff] [review] fennecbench.manifest 1.0 This adds fennecbench.manifest to the tpan/fennecmark directory. This will be inline with the instructions that build.sh display.
That's checked in, and tests are running.
verfied with beta3. There is a bug tracking the issues with the latest updates.