386995 - Add Tdhtml to Talos

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Schroepfer]

The Tdhtml test:

See URL and:

http://lxr.mozilla.org/mozilla/source/testing/tinderbox-standalone-tests/Tests/DHTMLPerformanceTest.pm#29

Needs to be added to Talos.

Comment 1

[alice nodelman [:alice] [:anode]]

Attachment: integrates dhtml/svg/gfx tests into talos

From the work that vlad did to make the existing dhtml/gfx/svg tests work with the pageloader, I've made the integration of these tests into talos.

I'm currently not doing any memory/cpu monitoring while the tests are run, simply recording the time that each tests takes to run.

Works on windows and mac, linux has not yet been tested.

This patch includes just the changes made to talos, more attachments to follow.

Comment 2

[alice nodelman [:alice] [:anode]]

The tests themselves don't make much sense as a patch (full of images, also huge) so I created a zip at 

http://people.mozilla.org/~anodelman/new-tests.zip

unzip in page_load_test directory to run the tests.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 282336 [details] [diff] [review]
integrates dhtml/svg/gfx tests into talos

r+ to this.

I'm not sure I agree about keeping the tests out of the tree. As long as there isn't a really compelling reason to keep them out (such as copyright issues like with the pageset) I think they should be put into the tree. Keeping them outside puts an additional burden on configuration. For Buildbot setup and standalone Talos I think the fewer places to download/checkout from, the better. As it stands now Talos must be checked out and a the pageset installed. Adding a "download x" requirement is going to be a pain.

I'm not too familiar with the nature of the tests, whether they are Talos specific or not, etc. So, there may be a reason to keep them out of Talos that's gone over my head.

Comment 4

[alice nodelman [:alice] [:anode]]

Ah - I've gone and been confusing.  Yes, the tests will be checked in to the tree in talos/page_load_test.  I didn't want to submit them as a patch because they aren't really a change set and they are also full of binary files (images and such).

Comment 5

[[e-mail] (:bhearsum)]

Ah, ok. Well, everything sounds good to me then!

Comment 6

[alice nodelman [:alice] [:anode]]

Attachment: adds -tprender switch to tgfx test, otherwise the same as before

The -tprender flag is required for the correct execution of the tgfx test.

Comment 7

[alice nodelman [:alice] [:anode]]

Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.12; previous revision: 1.11
done
Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.8; previous revision: 1.7
done

Also checked in
page_load_test/dhtml
page_load_test/svg
page_load_test/gfx

Comment 8

[alice nodelman [:alice] [:anode]]

There's a problem with the -tprender flag throwing off how the data from the browser is examined as it adds a 'Render report' to the end of the pageloader results.  This causes an exception and crashes talos.

Comment 9

[alice nodelman [:alice] [:anode]]

Attachment: make talos ignore incorrectly formatted browser data

This should fix the -tprender issue in combination with bug 397831

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 282617 [details] [diff] [review]
make talos ignore incorrectly formatted browser data

>-        for mypage in page_results[2:]:
>+        for mypage in page_results:

Why are we ignoring the first two page_results now?

Other than that, looks fine.

Comment 11

[alice nodelman [:alice] [:anode]]

The first two lines are of the format:

    report += "_x_x_mozilla_page_load,"+avgmed+",NaN,NaN\n";
    report += "_x_x_mozilla_page_load_details,avgmedian|"+avgmed+"|average|"+avg.toFixed(2)+"|minimum|NaN|maximum|NaN|stddev|NaN";

They do not contain information that we are interested in.  For the render report there is an additional "***** Render report *****" that also needs to be ignored.  With the new solution I just ignored all lines that do not have numerical results in them - instead of having to code in specific exceptions for specific formats.

Comment 12

[alice nodelman [:alice] [:anode]]

Checking in run_tests.py;
/cvsroot/mozilla/testing/performance/talos/run_tests.py,v  <--  run_tests.py
new revision: 1.13; previous revision: 1.12
done