Better integrate benchmark tests with our test framework

RESOLVED WONTFIX

Status

defect
RESOLVED WONTFIX
6 years ago
4 months ago

People

(Reporter: jimm, Unassigned)

Tracking

Details

(Whiteboard: p=0)

(Reporter)

Description

6 years ago
spun off from bug 859155:

(In reply to Jim Mathies [:jimm] from comment #23)
> (In reply to Sam Foster [:sfoster] from comment #22)
> > Comment on attachment 760857 [details] [diff] [review]
> > test set v.1
> > 
> > I dont feel like I (yet) have all the context I need to give a useful review
> > here, so please accept some drive-by comments. I understand that this gets
> > us an easy way to measure and track some metro-specific performance metrics
> > over time with (vs. talos which know little about tbh). To that end, minus
> > the grumbling below, this seems direct and pragmatic and we should start
> > hooking something up asap to see how it plays out. Provided we evaluate each
> > perf test to assure ourselves that it will provide meaningful insights (not
> > just data for its own sake), I tend to think some data is better than none
> > and its easier to iterate on a thing that's working. 
> > 
> > That said, I find the nesting of test syntaxes odd. We have test() and
> > gTest.push({... }) doing one thing, with the ok/is/etc. assertions we know
> > and love, then the PerfTest.declareTest apparently declaring a test inside a
> > test. Its just a naming oddness, but maybe one we can tuck away in our
> > helpers so we can at least interact with a more intuitive API, and start
> > establishing cow-paths. 
> 
> Yes it does seem somewhat hackish. I can ingrain these tests into our
> testing logic in head.js more. We have gTests that have methods on them that
> do specific things, we could have a gBenchmarks that have a similar
> execution model. Test info/output would be available via methods on the test
> object and spit out by the logic that controls test runs. Results could be
> returned in a tear down type method. head.js could also stand to be broken
> up into a shared library of test helpers and two shims that execute two
> different types of tests. 
> 
> Am I headed in the right direction here? 
> 
> I'd still like to get these landed as is and work on something like this in
> a follow up when I have time over the next week or so. 
> 
> > Are there perf assertions we want to make, or is gathering this data just a
> > side-effect for later perusal? There are some basic, default thresholds we
> > could plug in that would fail a test if it took too long. E.g. feedback on a
> > button click (toggling :active) should take < 100ms. Similarly with
> > immediate consequence of any user action - like focusing a field, dragging
> > to highlight text and so on. More than .1 second (roughly) tends to break
> > the illusion of direct manipulation; we should indicate something, even if
> > we can't always complete the outcome of the action in so short a time.
> 
> Yep, if we find something like that useful. Although we probably wouldn't
> tie test failures to something as specific as a button state change, I think
> we'd end up with more random orange than we could handle. But I can see us
> setting a min/max on the composition test in here, so that we could pick up
> serious regressions in gfx. 
> 
> > For perf data, we may want to log std deviation, to know if the average is
> > meaningful at all? 
> 
> Don't see why not. This could be spit out with the other data and graphed
> along side other data points.
> 
> > Out of the corner of my eye, I'm watching bug 867742, which is a proposal
> > for a new testsuite module. Its not relevant here while we are
> > experimenting, just something to keep in mind as our own automating testing
> > needs take shape.
> 
> If and when there's a push to better standardize mozilla in-tree testing
> we'll be a part of it.
Summary: Better integrate benchmark tests with our test framework → Story - Better integrate benchmark tests with our test framework
Whiteboard: feature=story c=testing u=developer p=0
(Reporter)

Comment 1

6 years ago
This is, when we can get to it. Not blocking v1 or any other release.
No longer blocks: metrov1backlog
Blocks: metrobacklog
No longer blocks: metro-testing
Summary: Story - Better integrate benchmark tests with our test framework → Better integrate benchmark tests with our test framework
Whiteboard: feature=story c=testing u=developer p=0 → [story]
(Reporter)

Comment 2

5 years ago
moving [story] bugs over to tracking product.
Component: Tests → Metro Operations
OS: Windows 8 Metro → Windows 8.1
Product: Firefox for Metro → Tracking
Version: Trunk → ---
Whiteboard: [story] → p=0
We never shipped the metro support, closing!
Status: NEW → RESOLVED
Last Resolved: 11 months ago
Resolution: --- → WONTFIX

Updated

4 months ago
Product: Tracking → Tracking Graveyard
You need to log in before you can comment on or make changes to this bug.