Draft test plan for the various search features

REOPENED
Unassigned

Status

()

REOPENED
2 years ago
6 months ago

People

(Reporter: gfritzsche, Unassigned)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(firefox52 wontfix)

Details

(Reporter)

Description

2 years ago
From recent bug findings we see that quality coverage around search (especially search Telemetry) was not sufficient so far.

This should come down to:
* going through our different search access points / sources in the UI (awesomebar, search bar, newtab, home, contextmenu, ...)
* for those, going through the possible actions / types (enter, one off, keyword, ...)
* accounting for default, shipped & custom search engines
* specifying the expected results (triggered search results in browser, in Telemetry SEARCH_COUNTS & engagement measures)

This might also be a good opportunity to add/update in-tree documentation from what we find (either search or telemetry specific docs, bug 1309256 is looking at getting the browser telemetry doc started).

Panos team can raise this with QA once the draft plan stands.
(Reporter)

Comment 1

2 years ago
Potentially helpful prior art is the test plan for bug 1304027:
https://public.etherpad-mozilla.org/p/awesomebar-regression
(Reporter)

Comment 2

2 years ago
Javauns metrics wishlist could also be a good starting point for the format:
https://docs.google.com/spreadsheets/d/1gDQbcKkMC1qj75DRsN5t5IkRs5oWMUxRYUnXflybJWw/
Priority: P2 → P1
Assignee: nobody → alessio.placitelli
Marco, does the format of the document from comment 3 look reasonable to you?
Status: NEW → ASSIGNED
Flags: needinfo?(mak77)
Does "Enter" also imply click? And well, the same is valid for most of the other entries. Maybe we should split each entry by keyboard/mouse, I'm not sure we have the same test coverage for the 2 interactions.
And also for telemetry, I don't think the current histograms distinguish keyboard and mouse, something UITelemetry does instead and it could be useful from a UX point of view. We may count that as a miss. But regardless, we should likely track status for both separately.

why does it state "open the result in the current tab", it can actually also open in a different tab or window.

re: System search, I'm not sure if it's there to track Cortana searches, but afaict MS removed any possibilities to do those.
Maybe you could replace that with activity stream, not sure.

I don't know if we want already to introduce future possible splittings, like search suggestions may be local or remote (in searchbar, home, newtab), we may want to track those separately.

I don't see anything about the search settings button that is part of one-off experience, do we plan to track it anywhere? I think currently it's only tracked by UITelemetry.

Apart from that, the tabular setup looks ok
Flags: needinfo?(mak77)
(In reply to Marco Bonardo [::mak] from comment #5)
> Does "Enter" also imply click? And well, the same is valid for most of the
> other entries. Maybe we should split each entry by keyboard/mouse, I'm not
> sure we have the same test coverage for the 2 interactions.
> And also for telemetry, I don't think the current histograms distinguish
> keyboard and mouse, something UITelemetry does instead and it could be
> useful from a UX point of view. We may count that as a miss. But regardless,
> we should likely track status for both separately.

Yes, that's basically indicating the "simple search" you can perform by writing in the awesomebar/seachbar and then simply pressing enter or clicking.

The current histograms/scalars don't make any distinction on the triggering input device. Feel free to add/edit the document as you like!

> why does it state "open the result in the current tab", it can actually also
> open in a different tab or window.

Whops, by bad, removed that.

> re: System search, I'm not sure if it's there to track Cortana searches, but
> afaict MS removed any possibilities to do those.
> Maybe you could replace that with activity stream, not sure.

Mh, that's for tracking [1], on which I've accidentally stumbled upon and I'm *assuming* it's for system search/Cortana. Unfortunately, I have no clue. Since the code is still there, I'll leave it on the spreadsheet with a note. We can always remove the entry if we remove the code.

> I don't know if we want already to introduce future possible splittings,
> like search suggestions may be local or remote (in searchbar, home, newtab),
> we may want to track those separately.
> 
> I don't see anything about the search settings button that is part of
> one-off experience, do we plan to track it anywhere? I think currently it's
> only tracked by UITelemetry.

I agree this should probably be covered by QA efforts, as other things that I'm probably not capturing in the spreadsheet because I don't know about them.

Once I get this structure going, I think the best path forward would be for you or :past (or somebody with a greater knowledge of the search area) to help filling in the gaps.

[1] - https://dxr.mozilla.org/mozilla-central/rev/8103c612b79c2587ea4ca1b0a9f9f82db4b185b8/browser/components/nsBrowserContentHandler.js#246
Panos, we filled the spreadsheet from comment 3! Can you take it from here?
Flags: needinfo?(past)
Yes, thank you for putting this together!
Assignee: alessio.placitelli → past
Flags: needinfo?(past)
(In reply to Panos Astithas [:past] from comment #8)
> Yes, thank you for putting this together!

My pleasure!
Whiteboard: [measurement:client] → [measurement:client:tracking]
I filed a number of implementation bugs blocking bug 1334599, so I think the work this bug was supposed to track is now completed. Feel free to reopen if I misunderstood.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 11

2 years ago
Are the gaps in test coverage filled now or do we have bugs filed for them?
Are the open questions in the document resolved or planned to be addressed in a separate bug?
Is there a plan to have QA work through the expected behavior in the doc to make sure the search features behave as intended?
Flags: needinfo?(past)
(In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> Are the gaps in test coverage filled now or do we have bugs filed for them?

The implementation bugs blocking bug 1334599 will come with automated tests. Are there any other gaps to fill?

> Are the open questions in the document resolved or planned to be addressed
> in a separate bug?

It's not clear which document you are referring to, but I tried to distill both spreadsheets (Javaun's and Alessio's) to implementation bugs. So I believe I incorporated every question in the spreadsheet comments.

> Is there a plan to have QA work through the expected behavior in the doc to
> make sure the search features behave as intended?

Yes, the plan is to have QA formulate a test plan based on both spreadsheets and bugs on file, once we have made more progress into the implementation.
Flags: needinfo?(past)
(Reporter)

Comment 13

2 years ago
(In reply to Panos Astithas [:past] from comment #12)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> > Are the gaps in test coverage filled now or do we have bugs filed for them?
> 
> The implementation bugs blocking bug 1334599 will come with automated tests.
> Are there any other gaps to fill?

Alessio?
Flags: needinfo?(alessio.placitelli)
(In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> (In reply to Panos Astithas [:past] from comment #12)
> > (In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> > > Are the gaps in test coverage filled now or do we have bugs filed for them?
> > 
> > The implementation bugs blocking bug 1334599 will come with automated tests.
> > Are there any other gaps to fill?
> 
> Alessio?

Mh, I can't directly spot the test coverage for the in-content searches (i.e. the about:* sections in [1]). If that's covered by other bugs there, we should be ok (at least, considering what I wrote in [1]).

[1] - https://docs.google.com/spreadsheets/d/1T9_hUxQGg1OwN8UruqIlQKTePUS3BMnKlDctmYbhUbo/edit#gid=2096837827
Flags: needinfo?(alessio.placitelli)
Alessio you are right, I haven't filed bugs for about:home and about:newtab yet. We consider them part of the second phase of the project and wanted to complete phase one first. This is still in my TODO list, but I'll reopen this bug as an additional reminder and opportunity to provide updates.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Mass wontfix for bugs affecting firefox 52.
status-firefox52: affected → wontfix
(Reporter)

Updated

a year ago
Component: Telemetry → Search
Priority: P1 → --
Product: Toolkit → Firefox
(Reporter)

Updated

a year ago
Whiteboard: [measurement:client:tracking]
Assignee: past → nobody
You need to log in before you can comment on or make changes to this bug.