Closed Bug 1436827 Opened 6 years ago Closed 6 years ago

add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets)

Categories

(Testing :: Raptor, enhancement)

Product:
Testing
Component:
Raptor
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox63 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox63 --- fixed

People

(Reporter: jmaher, Assigned: rwood)

References

Details

(Whiteboard: [PI:July])

Attachments

(1 file)

Bug 1436827 - add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets);
59 bytes, text/x-review-board-request
igoldan
: review+
Details 
This document is 200 pages and takes quite a few seconds to load.  We should measure something like this as it is real world and much more complex than our existing pages:
https://docs.google.com/document/d/1EPSmGqm2r4Qq42B4t1VOYacjTlL0JVuC8JSlUvoIhss/edit#heading=h.z0bnqrs2ilzg
Whiteboard: [PI:March]
On this particular page, there are several points in time that are interesting, but for "simplicity" the first page's content should be considered the "hero element". We can start there, but once we can track multiple points it might be worth adding 3-4 more points as it's quite a complex page load.
Whiteboard: [PI:March] → [PI:April]
Blocks: 1454053
Whiteboard: [PI:April] → [PI:May]
Whiteboard: [PI:May] → [PI:June]
this has been in the queue for a while, we should context switch and make some progress here by adding to tp6:
* google docs
* google slides
* google sheets

ideally some examples of each of these recorded with mitmproxy and at the very least playing back with ttfnbp.  We can optimize hero elements in a followup.

:rwood, can you start on getting these pages this week?  It should be easy to generate and use the pages in tp6, not sure about tp6_heavy, tp6_webext, etc- maybe we can start small and do this for tp6 osx and hand off part 2 to be all tp6* on osx; then part 3 to be all platforms; and part 4 to be meaningful measurements.  Ideally this would start in raptor and not be part of talos :)
Flags: needinfo?(rwood)
Whiteboard: [PI:June] → [PI:July]
Yes, thanks!
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Component: Talos → Raptor
Flags: needinfo?(rwood)
Summary: considera a new tp6 test for a large google doc to measure pageload and other rendering/ready states → consider a new tp6 test for a large google doc to measure pageload and other rendering/ready states
For the google document, I'll use this one as noted above in the description:

https://docs.google.com/document/d/1EPSmGqm2r4Qq42B4t1VOYacjTlL0JVuC8JSlUvoIhss/edit#heading=h.z0bnqrs2ilzg

I'll see if I can add a hero element on the first place at the "1. INTRODUCTION" header element.

For google slides I'll use this (just taken from a slides template):

https://docs.google.com/presentation/d/1Ici0ceWwpFvmIb3EmKeWSq_vAQdmmdFcWqaiLqUkJng/edit?usp=sharing

And I'll see if I can add a hero element on the first slide.

For google sheets I'll use this (taken from a sheets template):

https://docs.google.com/spreadsheets/d/1jT9qfZFAeqNoOK97gruc34Zb7y_Q-O_drZ8kSXT-4D4/edit?usp=sharing

And try to add a hero element somewhere.

I'm going to record the new pages on OSX this time (last time I recorded on Win 10) as it's easier since that's my dev environment, and it really doesn't matter what platform they are recorded on (we use the same set for all platforms).
Ah, I found Tarek's code that he used to insert the hero element into a mitmproxy recording:

https://github.com/tarekziade/mitmflow
Summary: consider a new tp6 test for a large google doc to measure pageload and other rendering/ready states → add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets)
Blocks: 1449180
No longer blocks: 1454053
I made the mitmproxy recordings as noted above, and I also added a hero element to the blue 'Sign In' button on the top right, for each one (doc/sheets/slides).

I've also saved the html source of the new pagesets (as served out by mitmproxy during the tests) to our new perf-automation github pageset repo [1]. In case the bare html is needed for debugging (outside of mitmproxy/raptor).

[1] https://github.com/mozilla/perf-automation/tree/master/pagesets/mitmproxy/raptor-gdocs

Once they're uploaded to tooltool then I can go ahead and finish this patch to turn on the new raptor-gdocs test suite. :)
Depends on: 1477109
Note: This patch depends on fixes in Bug 1454053 (see https://bugzilla.mozilla.org/show_bug.cgi?id=1454053#c30) and won't be green on try until that bug lands and this is rebased.
Depends on: 1454053
(In reply to Robert Wood [:rwood] from comment #12)
> Note: This patch depends on fixes in Bug 1454053 (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1454053#c30) and won't be green
> on try until that bug lands and this is rebased.

I tested this on Try, with the unlanded changes from bug 1454053. But it seems gdocs tests is failing: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2bef85473bb108513ad2001584a190a22653f1de&filter-tier=1&filter-tier=2&filter-tier=3
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #13)
> 
> I tested this on Try, with the unlanded changes from bug 1454053. But it
> seems gdocs tests is failing:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=2bef85473bb108513ad2001584a190a22653f1de&filter-
> tier=1&filter-tier=2&filter-tier=3

Thanks Ionut! :) Yes it looks like the google doc test is timing it - understandable as it takes alot longer to load these ones. I'll update the allowed time / timeout when I rebase with the changes from bug 1454053.
I'm going to just use the fnbpaint measurement for now for this new test suite. Once Bug 1478057 is done (report a single mean for the entire page instead of each measurement type) then we can also turn on measuring the hero elements for each page also.
Comment on attachment 8993717 [details]
Bug 1436827 - add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets);

Removing review flag until I can rebase and re-test.
Attachment #8993717 - Flags: review?(igoldan)
Comment on attachment 8993717 [details]
Bug 1436827 - add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets);

https://reviewboard.mozilla.org/r/258410/#review266350

::: testing/raptor/raptor/tests/raptor-gdocs.ini:19
(Diff revision 6)
> +unit = ms
> +lower_is_better = true
> +alert_threshold = 2.0
> +page_timeout = 30000
> +
> +[raptor-firefox-google-docs]

In the other *.ini files from testing/raptor/raptor/tests dir, our naming convention adds the browser name to the end of the test configs.
Basically, write raptor-google-docs-firefox instead of raptor-firefox-google-docs and so on.
Comment on attachment 8993717 [details]
Bug 1436827 - add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets);

https://reviewboard.mozilla.org/r/258410/#review266350

> In the other *.ini files from testing/raptor/raptor/tests dir, our naming convention adds the browser name to the end of the test configs.
> Basically, write raptor-google-docs-firefox instead of raptor-firefox-google-docs and so on.

Ah, the benchmark tests do that, but raptor-tp6 has it this way - you're right they shoud be consistent. I'll change tp6 and gdocs to match the benchmarks and put the browser name at the end. Thanks! :)
Comment on attachment 8993717 [details]
Bug 1436827 - add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets);

https://reviewboard.mozilla.org/r/258410/#review266354

This can land after another round of Try tests. Thanks!
Attachment #8993717 - Flags: review?(igoldan) → review+
I notice that the taskcluster configs for Raptor also have the browser at the start of the name, not at the end [1]. I'm going to change those also so that they have the browser name at the end, to match the Raptor test names. I'll update that, push to try and just carry fw the r+.

[1] https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/648cd558ea6a
add new raptor 'gdocs' pageload test suite (for google docs/slides/sheets); r=igoldan
https://hg.mozilla.org/mozilla-central/rev/648cd558ea6a
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: