Open Bug 1470473 Opened Last year Updated 19 days ago

[meta] Add more basic perfherder JS tests

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: wlach, Unassigned)

References

(Depends on 2 open bugs)

Details

(Keywords: meta)

Attachments

(2 files)

We only have a very basic perfherder automated test running right now:

https://github.com/mozilla/treeherder/blob/37fad3d7a5f016d08fd732a0088830b7a83216b8/tests/selenium/test_perfherder.py

This just tests that add test data works, which is a bit of a no-op given that there is no test data in the database by default. This type of test catches a surprising amount of bugs by itself, but we could do even better with just a little bit more work. At minimum, a set of tests that does the following would be good:

* Test that the test chooser shows test data when loaded (one fake talos series should be enough here) and that selecting the graph view renders some data points.
* Test that the compare view works (two fake pushes with two fake perf datums should be sufficient)
* Test that the performance alerts view works (just one fake alert / alert summary should be enough)

These tests should verify that these actions don't result in console errors and that the expected dom elements load.

Some examples of adding fake performance data to a treeherder database during automated testing are here:

https://github.com/mozilla/treeherder/blob/master/tests/webapp/api/test_performance_data_api.py
https://github.com/mozilla/treeherder/tree/master/tests/perfalert

I am happy to provide mentorship and advice to anyone who wants to take it on.
Blocks: 1450044
See Also: → 1470453
I want to chip in and add some tests..
(In reply to Ramkumar from comment #1)
> I want to chip in and add some tests..

Great. If you (or anyone else) is interested in working on this bug, the first step is to get a copy of treeherder running on a local vagrant instance:

http://treeherder.readthedocs.io/installation.html#setting-up-vagrant

From there, the next step is to explore and understand the perfherder front end:

https://treeherder.mozilla.org/perf.html

From there, the information in the first comment should be sufficient to write the tests, for someone willing to do a bit of exploration of the code and ui in the link above. If you have any questions, please feel free to ask on irc.mozilla.org #treeherder.
Have set up vagrant, but finding difficult to run tests.

Getting unrecognised command error while running below one
vagrant ~/treeherder$ ./runtests.sh

http://irc.mozilla.org/ doesnt loads here in uk, anyway to chat on this?
(In reply to Ramkumar from comment #3)
> Have set up vagrant, but finding difficult to run tests.
> 
> Getting unrecognised command error while running below one
> vagrant ~/treeherder$ ./runtests.sh

Unclear what's going on but in general you need to follow the instructions very precisely when setting up vagrant. It's very easy to get into a state where things are only partially working, which seems like what has happened here. I would start over from scratch. If you see anything indicating an error while running the setup steps in the wiki, stop and debug it (or ask about it on IRC).

> http://irc.mozilla.org/ doesnt loads here in uk, anyway to chat on this?

You'll need an IRC client -- see https://wiki.mozilla.org/IRC - note that most people (including me) are away on weekends.

If you want to try to continue work on this before next week, you can try emailing me privately with a detailed description of the error you're seeing -- i'll reply if I have time.
:wlach This bug sounds like a typical PI request [1]. Maybe we should fill a form and email at pi-request@mozilla.com

[1] https://mana.mozilla.org/wiki/display/PI/PI+Request
Flags: needinfo?(wlachance)
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #5)
> :wlach This bug sounds like a typical PI request [1]. Maybe we should fill a
> form and email at pi-request@mozilla.com
> 
> [1] https://mana.mozilla.org/wiki/display/PI/PI+Request

This was my initial thinking too, then Ramkumar dropped by. :) I've been helping him get up and running with Treeherder behind the scenes -- let's see how it goes over the next week or so. If it proves to be too difficult or time consuming (this is not exactly what I'd call a 'good first bug'), we can try to get someone in the PI organization to take over.
Flags: needinfo?(wlachance)
(In reply to William Lachance (:wlach) (use needinfo!) from comment #6)
> (In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #5)
> > :wlach This bug sounds like a typical PI request [1]. Maybe we should fill a
> > form and email at pi-request@mozilla.com
> > 
> > [1] https://mana.mozilla.org/wiki/display/PI/PI+Request
> 
> This was my initial thinking too, then Ramkumar dropped by. :) I've been
> helping him get up and running with Treeherder behind the scenes -- let's
> see how it goes over the next week or so. If it proves to be too difficult
> or time consuming (this is not exactly what I'd call a 'good first bug'), we
> can try to get someone in the PI organization to take over.

Sounds like a plan!
:wlach Any updates here?
:wlach I worked with Selenium tests before and I may be able to help.

I will work today to get my env working and try to run the current test
Assignee: nobody → bebe
Added one test for:
* Test that the test chooser shows test data when loaded (one fake talos series should be enough here) and that selecting the graph view renders some data points.

We can't implement the last part. AFAIK selenium can't easily interact with canvas elements. Because of that "selecting the graph view renders some data points" might not be implemented.
Hey Florin! Awesome you're taking this on.

(In reply to Florin Strugariu [:Bebe] from comment #11)
> Added one test for:
> * Test that the test chooser shows test data when loaded (one fake talos
> series should be enough here) and that selecting the graph view renders some
> data points.
> 
> We can't implement the last part. AFAIK selenium can't easily interact with
> canvas elements. Because of that "selecting the graph view renders some data
> points" might not be implemented.

You could test at least part of this by just navigating to a url that will automatically cause the popup to be shown (via a "selected" parameter). e.g.:

https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,1659449,1,1&selected=mozilla-central,1659449,392229,609285258,1

That said, it would be nice to be able to test the graphs interaction as well. Oh well, maybe someday.
wlach added the popup check please review.
Flags: needinfo?(wlachance)
Thank you for the PR!

Will, is this all of the tests you were thinking of, or do there need to be more?
If the latter, should this bug be left open, or another be filed? :-)
(In reply to Ed Morley [:emorley] from comment #15)
> Thank you for the PR!
> 
> Will, is this all of the tests you were thinking of, or do there need to be
> more?
> If the latter, should this bug be left open, or another be filed? :-)

IMO we should add tests for the compare view and alerts as outlined in comment 0. I don't really have an opinion on whether that should be done as part of this bug or a followup! Whatever works for people.
Flags: needinfo?(wlachance)
I would keep this open. 

You will get a pull request for the next tests in a few days.
I'm turning this bug into a meta bug, so we can elaborate the tests better.
Summary: Add more basic perfherder selenium tests → [meta] Add more basic perfherder selenium tests
(Changing remaining metadata to match org triage guidance for meta bugs)
Assignee: fstrugariu → nobody
Keywords: meta
Priority: -- → P3

We're moving away from selenium tests in favor of react-testing-library per bug 1576966.

Summary: [meta] Add more basic perfherder selenium tests → [meta] Add more basic perfherder JS tests
No longer blocks: 1450044
You need to log in before you can comment on or make changes to this bug.