All users were logged out of Bugzilla on October 13th, 2018

Create new Talos suite: "deep browsing"/"normal browsing" Tp pageset

RESOLVED WONTFIX

Status

P3
normal
RESOLVED WONTFIX
10 years ago
6 years ago

People

(Reporter: joe, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [talos])

(Reporter)

Description

10 years ago
Right now, our Tp pageset (and whatever new Tp pagesets that we get, for example from bug 473821) simply browses the top N sites, front page by front page. This doesn't really reflect real browsing habits as we understand them, which involve linking from page to page within a site, and linking from one page to another, potentially a deep link.

This is obviously a lot of work, but would benefit the realness of our Talos numbers.
At first glace this looks a lot like bug 473821.

I think that the request is more for a tp test that better approximates actual user experience (ie, a profile that contains information about the pages being loaded, the pages loaded in a logical order, etc) and may go so far as to include link clicking.

I would think that this would end up being a totally different metric than the current tp (and what will become the new tp as the pageset is rebuilt).  This is more of new design for a new test than a request to alter the current test.

Putting into the 'Future' bucket for now.
Component: Release Engineering: Talos → Release Engineering: Future
I think that our work with dirty profiles is doing what was originally asked for in this bug (ie, better simulation of people's actual browsing experience).

If I'm missing something feel free to re-open.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → WONTFIX
(Reporter)

Comment 3

9 years ago
Dirty profiles is part of it, but the other part is actually creating a set of pages that more accurately reflects our users' experiences, so we know whether we regress performance on that.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Can I get more information then about how we would collect such a set?
(Reporter)

Comment 5

9 years ago
Test Pilot, maybe? Or instrumenting someone browsing some of the top100 and usig that. It doesn't have to be perfect, just better than "browse the front pages of the top100."
Yeah, instrument a few clicks into each of the top 100 over the course of a week, and use that clickstream.  Without that we're totally blind on things like the effectiveness of real-world caching behaviour.
Summary: create "deep browsing"/"normal browsing" Tp pageset → Create Talos suite: "deep browsing"/"normal browsing" Tp pageset
In order to make this an "official" new Talos suite request, can you please attach a package of sites that you want this for?  Alice has tools to pull the pages and groom them into a Tp pageset but we need more to go on here.
Summary: Create Talos suite: "deep browsing"/"normal browsing" Tp pageset → Create new Talos suite: "deep browsing"/"normal browsing" Tp pageset
This has more info to guide whoever takes this on:

https://wiki.mozilla.org/Testing/Talos/NewTests#Overview
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3

Comment 10

9 years ago
Who should we assign this to re: comment 7 and comment 8?
Whiteboard: [talos]
(In reply to comment #10)
> Who should we assign this to re: comment 7 and comment 8?

Moving this bug to Testing:General for suite development, as requested by bmoss, ctalbert. 

Once this new suite is developed, please file a separate bug in
mozilla.org/ReleaseEngineering to enable running the new suite on builds
automatically.
Component: Release Engineering → General
Product: mozilla.org → Testing
QA Contact: release → general
Version: other → unspecified
Moving to Talos component.
Component: General → Talos
This may end up being a dupe of bug 601798 - as there is a lot of similar discussion there.

Comment 14

6 years ago
To echo comment 13, I think is a request for a specific pageset.  We'll need something more concrete to go on, but when we make the next pageset this should be taken into account
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.