1078254 - Talos - Add tests for newtab page load time

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Avi Halachmi (:avih)]

Bug 1059558 is a meta bug for improving newtab load time and UX (speed-wise). We need a tool to test newtab pageload times to measure such improvements.

TART was already modified last week to help with bug 1059558 and includes several related tests (5 - opening a new tab without animation of about:blank, about:newtab and about:newtab with preload, and just switching to about:blank and about:newtab URIs).

This bug is about incorporating these changes into the TART repo such that they can be used offline for testing, and possibly also as part of the standard runs of talos.

jmaher, wlach, do you think the "normal" page load tests could measure the load time of about:newtab? also with e10s? could they also do that for about:blank? (as a reference for what's the fastest we could load a page).

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:avih, I am not sure I understand the question or could even answer this.  I don't believe we can measure loading about:blank.  I don't know if about:newtab has any method for determining if we have completed the loading of the tab.  If we have an event to listen on (similar to mozafterpaint), then we can use pageloader and figure out how to make it work with e10s.

Comment 2

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #1)
> ... I don't know if
> about:newtab has any method for determining if we have completed the loading
> of the tab.  If we have an event to listen on (similar to mozafterpaint),
> then we can use pageloader and figure out how to make it work with e10s.

Why would about:newtab be different than tp5o pages?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

do we have the option to insert javascript in about:newtab?  when about:newtab loads, do we get a "load" and "mozafterpaint" event?

Comment 4

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #3)
> do we have the option to insert javascript in about:newtab?

We do, but we don't want to.

> when about:newtab loads, do we get a "load" and "mozafterpaint" event?

It does get a MozAfterPaint, and I guess it gets a load event too.

However, at the patch which we used last week to measure newtab, the issue is that the load event arrives quickly, followed by possibly more than one paint event (because it displays the content asynchronously after it finishes loading).

So I wonder if the pageloader addon has a mode of "last paint after the load event, after which there are no more paint events (within a certain timeout)", which will cover the newtab async load system.

I guess not, but then again, this might be a better way to test about:newtab load time instead of within TART (which basically implement this exact system of "measure duration until last MozAfterMaint after load" at the patch which I didn't yet post here).

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

We don't have a method for getting the last mozafterpaint in pageloader.  It would be nice to see that, but I am not sure of the best method for implementing it.

Comment 6

[William Lachance (:wlach)]

Off the top of my head, I don't have anything to add to what jmaher has said already. I'd certainly be happy to look at extending pageloader for this kind of use case at some point though.

Comment 8

[Avi Halachmi (:avih)]

Tim, I started working on making this patch for talos, and it occurred to me that TART v1.7.4 (which we used last week and has the new tests) wouldn't work in talos, because it relies on someone clicking the "set pinned tabs" button and then waiting for Firefox to populate the images, which takes both a considerable amount of time and requires network access (which talos doesn't have).

How do we make it work in a talos environment? I.e how do we populate the thumbnails?

I now recall we had exactly the same issue when we started with TART, and back then we didn't find a viable solution, but now that we want to add a test specifically for newtab load times, not having the thumbs populated is a deal breaker...

Comment 9

[Avi Halachmi (:avih)]

Attachment: bug1078254.v1.patch

For reference, this is the patch we used to measure newtab load times during last week, cleaned up a bit and applied on top of the pending patch from bug 1078250 (make TART e10s compatible).

Changes at this patch:

- (UI) Add 2 buttons: Set the pinned tiles to a fixed set of top-15 Alexa sites (needs restart and then wait till the background thumbnailer fetches all the images), and a button to restore the original tiles.

- (UI) Display a median value of the results when running locally.

- When setting TART to repeat each test NN times locally, initialize the test only once instead before each repeated run (such that we can measure first newtab with preload, and then following newtab with preload which may have different values than the first).

- One new measurement function (measureLastPaint) which measures the last MozAfterPaint within a defined timeout, after triggering the test function.

- 5 new tests which use measureLastPaint:

1. about:blank in a new tab.
2. about:newtab in a new tab (no tab animation) without preload.
3. about:newtab in a new tab (no tab animation) with preload.
4. changing location to about:blank at the current tab.
5. changing location to about:newtab at the current tab (preload not used).


TODO:

- Find out how to pre-populate the thumbnails programmatically for talos.

- Make the new tests e10s compatible (currently only 2,3 above work in e10s).

- Try to also measure load event time (will allow to understand how long it then takes to render from the load event).

Comment 10

[Avi Halachmi (:avih)]

(In reply to Avi Halachmi (:avih) from comment #9)
> ...
> TODO:
> 
> - Find out how to pre-populate the thumbnails programmatically for talos.
> ...

What if I set the pinned tiles pref to 15 different 'data:' URIs, where each of those fills the screen with stuff using javascript only (so no network access), then wait for an event which says "all thumbnails captured", then start the tests?

Maybe also set some pref to make the thumbnails capture with minimal delay between captures, such that it could finish all the captures in few seconds?

Comment 11

[Avi Halachmi (:avih)]

Attachment: bug1078254.v1.part2.patch

Adds a page at TART which generates random content (shapes/colors) according to its URL. Modified the "set pinned tiles" button to set the tiles to 15 different variants of this page (i.e. chrome://tart/content/pinned.html?1[/2/3...] ), such that we can get as many different thumbnails as we want without network access.

I set the complexity of the content such that it's roughly similar to non-synthetic content on my system, which is ~100k for a PNG image at 640x360 (Nightly thumbs size on my system).

When loaded with the pinned tiles pref set to these pinned tiles URLs, once about:newtab is opened, it takes it less than 3s to populate all the thumbnails/tiles with CGI images on my system. I think that's reasonably quick.

We can set the pref in talos such that the browser opens with these pinned tiles URLs, and all we'll need is to open the newtab page and wait few seconds for the thumbnail service to capture all of them.

Tim, how can I verify that the thumbnails service doesn't have any more pages to capture in its queue?

Comment 12

[Avi Halachmi (:avih)]

Attachment: bug1078254.v1.part3.patch - assure thumbs for talos

This is kinda hackish, but it seems to be working, and I don't yet have a cleaner way to achieve this.

The goal here is to assure that once TART starts running its tests, all the tiles thumbnails are already captured.

My approach was that talos sets a pref extensions.tart.assureTartThumbs to true for tests which need this.

TART examines this pref, and if true, on every browser window load, it sets the pinned tiles URLs via a pref (to the page of patch part 2) and then opens and closes about:newtab. talos only loads one window, so effectively this runs once.

Since talos waits about 10 seconds after startup and before starting the tests, and since the tiles are set and the new tab is opened much earlier than that, and since it takes about 3s to capture all the thumbs, by the time the test starts we've had at least 5 seconds of idle CPU, so I think we're good.

We'll probably have to change this to something cleaner/more deterministic, but for now this achieves the first TODO item from comment 9. The other 2 items still need addressing.

Comment 13

[Avi Halachmi (:avih)]

Also, talos sets the window size such that, on my system, it's only big enough to show 6 tiles on the newtab page. Assuming it won't show more tiles on other systems (retina Macs?), capturing 6 instead of 15 thumbs would require just more than 1 seconds instead of ~3.

Comment 14

[Avi Halachmi (:avih)]

Attachment: bug1078254.part3.v2.patch

This replaces the hacky trigger to capture the thumbs (by opening a newtab and closing it) with BackgroundPageThumbs.captureIfMissing(aURL) for the tile URLs, if possible (else it falls back to the open/close newtab trick - e.g. on Firefox older than 26, because I'd like to assure the thumbs are captures even on older Firefox, where possible).

This is reasonably clean IMO.

Comment 15

[Avi Halachmi (:avih)]

I'm having very hard time making the tests at the patches so far also work with e10s. My approach has been to make the tests use framescript, test them with normal firefox, and if they work, test also with e10s.

They already don't work from a framescript with normal firefox.

This is what should be working with a framescript:

1. about:blank in a new tab (no tab animation).
2. about:newtab in a new tab (no tab animation) without preload.
3. about:newtab in a new tab (no tab animation) with preload.
4. changing location to about:blank at the current tab.
5. changing location to about:newtab at the current tab (preload not used).

I managed to capture MozAfterPaint events at the framescript on all cases except when preload is enabled (3 above), which is the most important case of them all, and without it there's probably not much point to these tests.

The following framescript code works for all the tests except #3:

addEventListener("MozAfterPaint", function (e) {
  if (!e.clientRects.length)
    return;
  console.log("Got MozAfterPaintEvent");
});

Clearly It must paint to screen at some stage, but either this paint doesn't generate the MozAfterPaint event when opening newtab with preload, or somehow it just doesn't get to the framescript handler.

What we can do, however, is land these tests (patches parts 1,2,3 which I posted already) for the "normal" non e10s Firefox, and just don't run them on e10s.

Tim, what do you think?

Comment 16

[Avi Halachmi (:avih)]

Attachment: bug1078254.part4.WIP.patch

For reference: Use framescript to grab the measurements (not working well).

Comment 17

[Ed Lee :Mardak]

If we aren't measuring thumbnail capture time, we can somewhat simulate a screen of 15 thumbnails with 15 directory tiles where the image is a saved thumbnail. This avoids the network calls.

The main downsides off the top of my head:
- changes to thumbnailer (e.g., appropriately sized thumbnails) aren't reflected
- down the line, directory tiles vs history tiles could be completely different visually (where right now the main difference between history and directory tile is the image source)

Comment 18

[Avi Halachmi (:avih)]

(In reply to Ed Lee :Mardak from comment #17)
> If we aren't measuring thumbnail capture time, we can somewhat simulate a
> screen of 15 thumbnails with 15 directory tiles where the image is a saved
> thumbnail. This avoids the network calls.
> 
> The main downsides off the top of my head:
> - changes to thumbnailer (e.g., appropriately sized thumbnails) aren't
> reflected
> - down the line, directory tiles vs history tiles could be completely
> different visually (where right now the main difference between history and
> directory tile is the image source)

This was solved at patches part 2 and part 3.

Part 2 set the pinned tiles to URLs within the TART addon: chrome://tart/content/pinned.html?1[/2/3../15] where each of those generates a different content that can be loaded and rendered without network access.

Part 3 makes it capture all of them (if a pref is set) before the test starts.

This runs all the capturing and rendering through existing "code pipes" without any kinds of custom hacks.

Comment 19

[Avi Halachmi (:avih)]

(In reply to Avi Halachmi (:avih) from comment #15)
> ...
> What we can do, however, is land these tests (patches parts 1,2,3 which I
> posted already) for the "normal" non e10s Firefox, and just don't run them
> on e10s.

Apparently this wouldn't work because jmaher doesn't want to run tests which are not e10s-compatible, because he says e10s should be turned on by default on m-c in a month or so, and from that point onwards we'll only run talos on e10s on m-c - so no point in running them for a very limited period till that happens.

So we're kinda stuck here as long as these test don't fully work on e10s.

One suggestion which came up is that the newtab page itself recognizes the point in time of "newtab is done loading and painting", and fires an event with these details, where this event will only be used for testing purposes, such that it's easier for TART to follow only a single event for its measurements, especially considering that preload, e10s, and general newtab changes take place and it will be much easier to maintain the "done now" code at the context of the newtab code rather than in TART.

^ This doesn't really solve the issue I mentioned at comment 15 and a solution will still need to be found, but it will at least decouple this issue from TART.

I'll file a bug on this.

Comment 20

[Tim Taubert [:ttaubert] (inactive)]

(In reply to Avi Halachmi (:avih) from comment #8)
> How do we make it work in a talos environment? I.e how do we populate the
> thumbnails?

I think the best way to achieve that would be to package the thumbnail in TART itself and copy those to the profile directory when the add-on is loaded. That would certainly be the easiest way without having to let the whole capture mechanism run. Also more future-proof because the add-on wouldn't be dependent on the capture implementation.

(In reply to Avi Halachmi (:avih) from comment #11)
> Tim, how can I verify that the thumbnails service doesn't have any more
> pages to capture in its queue?

You can't from the outside. A simple solution would be to set browser.pagethumbnails.capturing_disabled=false.

(In reply to Avi Halachmi (:avih) from comment #15)
> The following framescript code works for all the tests except #3:
> 
> addEventListener("MozAfterPaint", function (e) {
>   if (!e.clientRects.length)
>     return;
>   console.log("Got MozAfterPaintEvent");
> });

The problem is that you add the frame script to the current tab's docShell. After swapping docShells the current tab will have a new docShell that didn't load your frame script and thus doesn't send any message about paints. This might be easier with bug 1077652 or you could attach a frame script to all browsers (using getGroupMessageManager) and ignore the ones where document.hidden=true. Another idea might be to attach the frame script after docShells were swapped, it *should* have negligible perf impact.

(In reply to Avi Halachmi (:avih) from comment #19)
> One suggestion which came up is that the newtab page itself recognizes the
> point in time of "newtab is done loading and painting", and fires an event
> with these details, where this event will only be used for testing purposes,
> such that it's easier for TART to follow only a single event for its
> measurements, especially considering that preload, e10s, and general newtab
> changes take place and it will be much easier to maintain the "done now"
> code at the context of the newtab code rather than in TART.

I'm not a big fan of shipping code that only helps testing. Same for performance testing. I'm quite confident that we don't need this, frame scripts are a great way to add code at runtime if you ignore the docShell swapping troubles.

Comment 21

[Tim Taubert [:ttaubert] (inactive)]

Yeah, I won't get to this anytime soon, sorry.