728435 - develop Necko performance tests (for use with Stone Ridge)

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect)

Description

[Josh Aas]

We need to develop Necko performance tests for use with our automated Necko performance testing based on NeckoNet (Stone Ridge). We'll be sending output to a public graph server.

https://wiki.mozilla.org/Necko/Performance/AutomatedTesting

Comment 1

[u408661]

Honza, don't know how much info dump you got from the work week, but I thought I'd dump what our initial thoughts were here in case you haven't heard anything yet. Of course, since you're going to be writing most of the new code, you're welcome to make changes as you see fit.

The best idea we came up with (others can correct me if I'm wrong on my opinion of 'best') was to record regular browsing sessions (or simulated sessions) using web-page-replay. We can use the list of URLs plus timing information recorded by wpr to inform some xpcshell-driven thing of what channels to create and when (in relation to each other) to create them. We can then use that against neckonet serving the recorded responses to measure our timing on the various operations we care about (probably starting with just total time to load the channel is easiest). We wanted to make sure we cover HTTP, HTTPS, SPDY, WebSockets and XHR with these tests. Given that, we'll probably need to do a bit of work on wpr to handle HTTPS, SPDY and WebSockets, so we'll probably want to start with HTTP and XHR, and then expand as we add capabilities to wpr.

As far as I can remember, that's the most in-depth we got on this topic. Others can feel free to expand/correct me if I left something out or got something wrong :)

Comment 2

[Patrick McManus [:mcmanus]]

I updated that page last night with some suggestions on network profiles.

Comment 3

[Jason Duell]

> I updated that page last night with some suggestions on network profiles.

O rly?  Revision history shows only changes from josh, with last one on Feb 21st:

  https://wiki.mozilla.org/index.php?title=Necko/Performance/AutomatedTesting&action=history

Comment 4

[Patrick McManus [:mcmanus]]

crap.. I updated https://wiki.mozilla.org/Necko/Performance/AutomatedTesting_profiles

was that wrong.

I hate wikis :)

Comment 5

[Josh Aas]

(In reply to Patrick McManus [:mcmanus] from comment #4)
> crap.. I updated
> https://wiki.mozilla.org/Necko/Performance/AutomatedTesting_profiles

I moved that content to the main Stone Ridge page (URL in bug description).

Comment 6

[Honza Bambas (:mayhemer)]

Summary of what we have agreed with Nick recently:

- test harness will provide API to write the results
- we will have a utility script automatically included in the context of the test
- that will provide some common basic functionality
  - do_write_result("testname", startTime, endTime)
  - do_test_pending() // if necessary, although I expect all tests to be async
  - do_test_finish()
- host names will for the start be example.com/org/net directing to the test machine
- time measurement will be done using JS Date object that is implemented to support hi-res timers on all platforms
- SSL/TLS will be handled later
- profile directory will also be sooner or later needed for cache and SSL ; an API like do_get_profile() is the way here
- the harness also has to process exceptions thew by the test, on any thread, in any piece of code
- as well we need to support timeouts of tests ; my proposal is to let the harness do this, if we need to change the timeout or reset it, provide an API

The API that the harness will provide is now just a proposal, but I want to inspire in xpcshell tests.  We all are used to it, it works and is simple.  Also, we can change the test -> test harness "communication protocol" anytime w/o a mass change of existing tests.


Actually the first test file might look merely like:

/**
 * A simple request test.  This measures the time between creation of the request channel
 * and first delivery of the data and time needed to deliver all data expected plus an overall
 * time to complete the load.  The request must succeed otherwise the test fails.
 */

var startTime = null;
var firstDataTime = null;
var endTime = null;

listener = {
  onStartRequest() {},
  onDataAvailable() {
    if (!firstDataTime)
       firstDataTime = new Date();
  },
  onStopRequest(request, context, status) {
    do_check(status === 0, "Simple request succeeded");

    endTime = new Date();

    do_write_result("Simple request latency", startTime, firstDataTime);
    do_write_result("Simple request data delivery", firstDataTime, endTime);
    do_write_result("Simple request completed", startTime, endTime);
    do_test_finish();
  }
};

function test() // The entry point
{
  startTime = new Date();
  var channel = make_channel("http://example.org/"); // make_channel func should also be provided
  channel.asyncOpen(listener, null);

  do_test_pending();
}

Comment 7

[u408661]

This all looks good, though I propose we do away with do_test_pending. We can just assume all tests are asynchronous, and so we should just call do_test_finished() once *all* the tests in the file have completed, whether they're async or not. The harness will just expect things to behave that way.

Comment 8

[u408661]

I've been thinking about this in relation to the graph server integration, and I think we might want to make the keys (your first argument to do_write_result) something that can be re-used across different tests. So for example, in each test we could do something like

do_write_result("latency", startTime, firstDataTime);
do_write_result("data_delivery", firstDataTime, endTime);
do_write_result("total", startTime, endTime);

and maybe we have a do_name_test("Simple Request") which is called at the start of the test so we know what test this is (we could also just determine this based on the name of the file).

This will make it easier to break the results from different tests into similar groupings, which will make massaging the data into something for the graphserver much more reliable.

Thoughts?

Comment 9

[u408661]

I should note, that it won't be necessary for each test to use exactly the same set of keys, if they don't make sense for that test. We should just have a defined set of keys (that can be extended) for simplicity's sake.

Comment 10

[Honza Bambas (:mayhemer)]

(In reply to Nick Hurley [:hurley] from comment #8)
> and maybe we have a do_name_test("Simple Request") which is called at the
> start of the test so we know what test this is (we could also just determine
> this based on the name of the file).

Yep.  I like it!  I had a similar thought too.

Other option is to have a manifest, similar to what xpcshell test has, that will list the files and give them screen names (more proper way IMO).

Comment 11

[Honza Bambas (:mayhemer)]

On the other hand, just the file name of the test may do as well and it's simpler to search for it when people want to see what the test is doing.

Comment 12

[u408661]

OK, let's just go with using the file name for now, and just make it a review requirement for any new stone ridge test suite to have a descriptive filename. If we need to add something like a manifest (or just a mapping from file -> descriptive name) later on, it shouldn't be difficult, but I like keeping this simple for now.

Comment 13

[Honza Bambas (:mayhemer)]

Attachment: preview 1

So, this the test from comment 6, slightly adjusted.

make_channel function is in the test for initial simplicity, but it has to be moved to a harness file that is automatically loaded in xpcshell along with a test.  We may get inspired from head_channels.js file.

Comment 14

[u408661]

Awesome, that looks great. Having make_channel in the harness should be no problem. Any other utility functions you think are desirable, just let me know, and we can get them in the harness. That's the nice thing about building this from the ground up, we can put in whatever we want without having anyone else not wanting us to :)

Comment 15

[Honza Bambas (:mayhemer)]

Stone Ridge is dead.