474338 - Create new Talos suite: "deep browsing"/"normal browsing" Tp pageset

Status: RESOLVED WONTFIX

(Testing :: Talos, defect, P3)

Description

[Joe Drew (not getting mail)]

Right now, our Tp pageset (and whatever new Tp pagesets that we get, for example from bug 473821) simply browses the top N sites, front page by front page. This doesn't really reflect real browsing habits as we understand them, which involve linking from page to page within a site, and linking from one page to another, potentially a deep link.

This is obviously a lot of work, but would benefit the realness of our Talos numbers.

Comment 1

[alice nodelman [:alice] [:anode]]

At first glace this looks a lot like bug 473821.

I think that the request is more for a tp test that better approximates actual user experience (ie, a profile that contains information about the pages being loaded, the pages loaded in a logical order, etc) and may go so far as to include link clicking.

I would think that this would end up being a totally different metric than the current tp (and what will become the new tp as the pageset is rebuilt).  This is more of new design for a new test than a request to alter the current test.

Putting into the 'Future' bucket for now.

Comment 2

[alice nodelman [:alice] [:anode]]

I think that our work with dirty profiles is doing what was originally asked for in this bug (ie, better simulation of people's actual browsing experience).

If I'm missing something feel free to re-open.

Comment 3

[Joe Drew (not getting mail)]

Dirty profiles is part of it, but the other part is actually creating a set of pages that more accurately reflects our users' experiences, so we know whether we regress performance on that.

Comment 4

[alice nodelman [:alice] [:anode]]

Can I get more information then about how we would collect such a set?

Comment 5

[Joe Drew (not getting mail)]

Test Pilot, maybe? Or instrumenting someone browsing some of the top100 and usig that. It doesn't have to be perfect, just better than "browse the front pages of the top100."

Comment 6

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Yeah, instrument a few clicks into each of the top 100 over the course of a week, and use that clickstream.  Without that we're totally blind on things like the effectiveness of real-world caching behaviour.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

In order to make this an "official" new Talos suite request, can you please attach a package of sites that you want this for?  Alice has tools to pull the pages and groom them into a Tp pageset but we need more to go on here.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

This has more info to guide whoever takes this on:

https://wiki.mozilla.org/Testing/Talos/NewTests#Overview

Comment 9

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 10

[Aki Sasaki (not active)]

Who should we assign this to re: comment 7 and comment 8?

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #10)
> Who should we assign this to re: comment 7 and comment 8?

Moving this bug to Testing:General for suite development, as requested by bmoss, ctalbert. 

Once this new suite is developed, please file a separate bug in
mozilla.org/ReleaseEngineering to enable running the new suite on builds
automatically.

Comment 12

[alice nodelman [:alice] [:anode]]

Moving to Talos component.

Comment 13

[alice nodelman [:alice] [:anode]]

This may end up being a dupe of bug 601798 - as there is a lot of similar discussion there.

Comment 14

[Jeff Hammel]

To echo comment 13, I think is a request for a specific pageset.  We'll need something more concrete to go on, but when we make the next pageset this should be taken into account