Closed Bug 1577905 Opened 3 months ago Closed 2 months ago

Support warmload tests under --browsertime

Categories

(Testing :: Raptor, enhancement, P2)

Version 3
enhancement

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: nalexander, Assigned: tarek, NeedInfo)

References

(Blocks 5 open bugs)

Details

Attachments

(1 file)

Out of the box, browsertime supports cold pageload tests. This ticket tracks extending to warm pageload tests. The way this works in browsertime is that one runs a Selenium Node.js script to drive the additional loads.

nalexander and acreskey both have versions of this.

Priority: -- → P2

Tarek: it's probably worth implementing this so that we can run more cycles, more quickly, while we align the two harnesses. (Otherwise we wait a long time for each cold load run to start.)

Locally this runs through a Node.js script, like:

// From https://github.com/acreskeyMoz/browsertime_tools/blob/ecde1ad8f55fdbc6be95188481951f05a07aaa85/reload.js.
module.exports = async function(context, commands) {
    let url = context.options.perftest.url;
    let reloads = context.options.perftest.reloads;
    await commands.measure.start(url, 'cold load');

    // According to the docs you need to prepend a dummy parameter to test the same page in browsertime:
    // https://www.sitespeed.io/documentation/sitespeed.io/scripting/
    let dummy = url.indexOf('?') == -1 ? '/?dummy' : '&dummy';
    for (let count = 0; count < reloads; count++) {
        await commands.wait.byTime(3000);
        await commands.measure.start(url + dummy + count, 'reload' + count);
    }
    return true;
};

Then we'll do something like

bin/browsertime.js ... warmload.js --perftest.url='https://example.com' --perftest.reloads=5

I haven't tested this locally in a long, long time so you'll have to investigate.

Notes:

  1. Raptor has browser_cycles and page_cycles. This will be page_cycles.
  2. Bonus points for handling both browser and page cycles.
  3. You can differentiate coldload from warmload with test.get('cold').
  4. I expect that result processing will need to accommodate differently shaped browsertime.json due to only having one Browsertime invocation with multiple results in the single run.
Flags: needinfo?(tarek)

I believe that the dummy parameters interfere with Necko caching, but I'm not aware of what the best option is here. acreskey, do you have an opinion on what we should do for warmload testing?

Flags: needinfo?(acreskey)

I'm not sure why the sitespeed.io documentation says "testing the same site multiple times will break".

I had started off with the dummy url method but I'm sure it does break Necko caching and it does indeed produce a differently-shaped browsertime.json which isn't great.

So I went to a method that's closer to what Denis was using:
(Transcribed here with the automation variables)

module.exports = async function(context, commands) {
    let url = context.options.perftest.url;
    let reloads = context.options.perftest.reloads;
    
    await commands.navigate('https://www.example.com');
    await commands.wait.byTime(30000); // magic number
    
    // Initial cold load and wait
    await commands.navigate(url);
    await commands.wait.byTime(10000); // magic number

    // Warm loads
    for (let count = 0; count < reloads; count++) {
        await commands.wait.byTime(3000);  // magic number
        await commands.measure.start(url);
    }
    return true;
};

This one is throwing away the results of the cold load.

The browsertime.json shape is identical to the cold load, so the process.py and R scripts all work out of the box.

Flags: needinfo?(acreskey)

Support warmload tests under --browsertime

Assignee: nobody → tarek
Flags: needinfo?(tarek)

Nick, here's a minimal diff that runs multiple page cycles and grabs them in the results - seems to work. I think we can have a single bt script for both cold/warm tests, but we should talk about it before I go further here.

Flags: needinfo?(nalexander)
Attachment #9093864 - Attachment description: Bug 1577905 - Support warmload tests under --browsertime r?nalexander [DO NOT MERGE YET] → Bug 1577905 - Support warmload tests under --browsertime r?nalexander,rwood
Blocks: 1581990
Blocks: 1585010
Pushed by tziade@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9eb8f5738e31
Support warmload tests under --browsertime  r=nalexander,rwood
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Blocks: 1582756

This landed! Clearing NI.

Flags: needinfo?(nalexander)

I must have made a mistake in Comment 3 -- although that script works for warm loads the shape of the json is quite different.
The cold load format has all the iterations within the first element's browserscripts.
This warm load format creates new root elements for each warm load.

Denis, did you happen to have a warm load script that gives browsertime.json in the same shape as the default cold loads?
I was experimenting with the measure command but wasn't successful.

Flags: needinfo?(dpalmeiro)
You need to log in before you can comment on or make changes to this bug.