Closed Bug 1355726 Opened 7 years ago Closed 7 years ago

Talos test for WebGL video texture upload

Categories

(Testing :: Talos, enhancement)

Product:
Testing
Component:
Talos
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox55 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla55
Tracking Flags:
Tracking Status
firefox55 --- fixed

People

(Reporter: daoshengmu, Assigned: daoshengmu)

References

Details

Attachments

(1 file)

Bug 1355726 - Add a talos test for WebGL video texture upload;
59 bytes, text/x-review-board-request
jgilbert
: review+
jmaher
: review+
Details 
We need to add a talos test to make sure our WebGL video texture upload always be run at the fast-path.
Assignee: nobody → dmu
Component: Canvas: WebGL → Talos
Product: Core → Testing
Comment on attachment 8857360 [details]
Bug 1355726 - Add a talos test for WebGL video texture upload;

https://reviewboard.mozilla.org/r/129340/#review132022

::: testing/talos/talos/test.py:427
(Diff revision 2)
> +    tploadnocache = True
> +    tpmozafterpaint = False
> +    gecko_profile_interval = 2
> +    gecko_profile_entries = 2000000
> +    win_counters = w7_counters = linux_counters = mac_counters = None
> +    """ ASAP mode """

Remove this line too.

::: testing/talos/talos/test.py:428
(Diff revision 2)
> +    # preferences = {'layout.frame_rate': 0,
> +    #                'docshell.event_starvation_delay_hint': 1,
> +    #                'dom.send_after_paint_to_content': False}

Remove these commented lines.

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:9
(Diff revision 2)
> +<html>
> +<head>
> +<script src="chrome://talos-powers-content/content/TalosPowersContent.js"></script>
> +<script language="javascript" type="text/javascript">
> +var start = 0;
> +var elapsedTime = 0;

This is unused now, so remove it.

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:17
(Diff revision 2)
> +var video;
> +video = document.createElement('video');
> +video.src = "../../../video/clips/testsrc.1080p.60fps.mp4";
> +video.addEventListener('playing', playingListener, true); // loadstart?
> +video.loop = true;
> +var kTotalFrames = 100;

kTotalTicks

Also, global constants should go above global variables, so move this above start/gl/video.

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:34
(Diff revision 2)
> +  video.removeEventListener('playing', playingListener, true);
> +  start = performance.now();
> +  tick();
> +}
> +
> +var frameCount = 0;

tickCount

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:39
(Diff revision 2)
> +  if (frameCount > kTotalFrames) {
> +    measurementEnded();
> +    return;
> +  }

Move this to the top of the function, before texImage2D.
This way, we always get one full tick: gl.texImage2D+setTimeout(0).

Change > to >=, as well. (consider kTotalTicks=0, and =1)

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:48
(Diff revision 2)
> +
> +  window.setTimeout(tick, 0);
> +}
> +
> +function measurementEnded() {
> +  reportResult(performance.now() - start);

var elapsed = performance.now() - start;
reportResult(kTotalTicks, tickMeanTime);

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:51
(Diff revision 2)
> +function reportResult(elapsedTime) {
> +  var msg = '\n----- WebGL Video texture upload ------\n';
> +  msg += 'Elapsed time: ' + elapsedTime + ' ms/' + kTotalFrames + ' frames\n\n';
> +
> +  dump(msg); // Put the readable report at talos run-log

function reportResult(totalTime, totalTicks) {
  var meanTickTime = totalTime / totalTicks;

  var desc = 'Mean tick time across ' + totalTicks + ' ticks: ';
  var text = desc + meanTickTime + 'ms';

  dump('[talos glvideo result] ' + text);

::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:59
(Diff revision 2)
> +
> +  dump(msg); // Put the readable report at talos run-log
> +
> +  if (window.tpRecordTime) {
> +    // Within talos - report the results
> +    return tpRecordTime(elapsedTime, 0, 'Elapsed time: ');

tpRecordTime(meanTickTime, 0, desc);
Attachment #8857360 - Flags: review?(jgilbert)
can we put this in the g4 job instead of g1- that job is running faster than g1.
(In reply to Jeff Gilbert [:jgilbert] from comment #5)
> Comment on attachment 8857360 [details]
> Bug 1355726 - Add a talos test for WebGL video texture upload;
> 
> https://reviewboard.mozilla.org/r/129340/#review132022
> 
> ::: testing/talos/talos/tests/webgl/benchmarks/video/video_upload.html:59
> (Diff revision 2)
> > +
> > +  dump(msg); // Put the readable report at talos run-log
> > +
> > +  if (window.tpRecordTime) {
> > +    // Within talos - report the results
> > +    return tpRecordTime(elapsedTime, 0, 'Elapsed time: ');
> 
> tpRecordTime(meanTickTime, 0, desc);


Try again, https://treeherder.mozilla.org/#/jobs?repo=try&revision=54c91d8cadc487067145f99302db83af6bd88ad5.

If we choose to use meanTickTime as our samples, the stats result would be quite small, like [#0] 867.8550000000005 ticks:   Cycles:1  Average:0.02  Median:0.02  stddev:0.00 (0.0%).
The new mean numbers should just be the previous number/100, so not too small. Double-check the results, because it should be in the 3-30 milliseconds range, more or less.
Comment on attachment 8857360 [details]
Bug 1355726 - Add a talos test for WebGL video texture upload;

https://reviewboard.mozilla.org/r/129340/#review132022

> kTotalTicks
> 
> Also, global constants should go above global variables, so move this above start/gl/video.

btw, I use `const` instead of `var`.
(In reply to Jeff Gilbert [:jgilbert] from comment #9)
> The new mean numbers should just be the previous number/100, so not too
> small. Double-check the results, because it should be in the 3-30
> milliseconds range, more or less.

Oops. I find my mistake. Now, It looks better.

08:55:42     INFO -  PID 49063 | ------- Summary: start -------
08:55:42     INFO -  PID 49063 | Number of tests: 1
08:55:42     INFO -  PID 49063 |
08:55:42     INFO -  PID 49063 | [#0] Mean tick time across 100 ticks:   Cycles:5  Average:57.11  Median:55.56  stddev:6.29 (11.3%)  stddev-sans-first:2.00
08:55:42     INFO -  PID 49063 | Values: 67.9  52.2  56.6  53.3  55.6
08:55:42     INFO -  PID 49063 | -------- Summary: end --------

Try link, https://treeherder.mozilla.org/#/jobs?repo=try&revision=37a28db37cad3378c5530c38fa58bcc1e4d8b17c
Comment on attachment 8857360 [details]
Bug 1355726 - Add a talos test for WebGL video texture upload;

https://reviewboard.mozilla.org/r/129340/#review132340

::: testing/talos/talos/test.py:418
(Diff revision 4)
>  
>  @register_test()
> +class glvideo(PageloaderTest):
> +    """
> +    WebGL video texture update with 1080p video.
> +    Measures time elapsed (ms) in 100 frames.

Measures mean tick time across 100 ticks.
(each tick is texImage2D(<video>)+setTimeout(0))
Attachment #8857360 - Flags: review?(jgilbert) → review+
Comment on attachment 8857360 [details]
Bug 1355726 - Add a talos test for WebGL video texture upload;

https://reviewboard.mozilla.org/r/129340/#review132470

I am doing some retriggers to look for stability and ensuring this runs fine on all platforms.  Unfortunately I cannot add linux64-stylo(g4) so we either need another try push or wish for good luck.

I would also like to ensure this test is documented on our wiki:
https://wiki.mozilla.org/Buildbot/Talos/Tests

this way when we point someone at a regression they have some idea about what this test is doing, why we are doing it, and what we are measuring.

so far I have windows 8 data, and it looks more bi-modal:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Btry,93f028f49dfc9f2bf20ca4105fb53f5ccffd5cc1,1,1%5D&series=%5Btry,828a180de562ea799c6106b14150c58d0998ac5f,1,1%5D

if the other platforms are more stable then I would be happier with this.
windows 7 is very stable!
Comment on attachment 8857360 [details]
Bug 1355726 - Add a talos test for WebGL video texture upload;

https://reviewboard.mozilla.org/r/129340/#review132536

marking as r+, please add wiki entry and work to make this stable if needed.
Attachment #8857360 - Flags: review?(jmaher) → review+
Pushed by dmu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f524336df20e
Add a talos test for WebGL video texture upload; r=jgilbert,jmaher
Blocks: 1322691
https://hg.mozilla.org/mozilla-central/rev/f524336df20e
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
I am trying to describe this test and the wiki link doesn't describe what the test is really doing.  Can you please update the test on the wiki so others can understand what perf we are measuring and what the results mean?
Flags: needinfo?(dmu)
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #21)
> I am trying to describe this test and the wiki link doesn't describe what
> the test is really doing.  Can you please update the test on the wiki so
> others can understand what perf we are measuring and what the results mean?

I have updated it, https://wiki.mozilla.org/Buildbot/Talos/Tests#glvideo. Please feel free to give feedback. Thanks.
Flags: needinfo?(dmu)
much better, thank you!
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: