1311800 - Draft test plan for the various search features

Status: RESOLVED WORKSFORME

(Firefox :: Search, task, P3)

Description

[Georg Fritzsche [:gfritzsche]]

From recent bug findings we see that quality coverage around search (especially search Telemetry) was not sufficient so far.

This should come down to:
* going through our different search access points / sources in the UI (awesomebar, search bar, newtab, home, contextmenu, ...)
* for those, going through the possible actions / types (enter, one off, keyword, ...)
* accounting for default, shipped & custom search engines
* specifying the expected results (triggered search results in browser, in Telemetry SEARCH_COUNTS & engagement measures)

This might also be a good opportunity to add/update in-tree documentation from what we find (either search or telemetry specific docs, bug 1309256 is looking at getting the browser telemetry doc started).

Panos team can raise this with QA once the draft plan stands.

Comment 1

[Georg Fritzsche [:gfritzsche]]

Potentially helpful prior art is the test plan for bug 1304027:
https://public.etherpad-mozilla.org/p/awesomebar-regression

Comment 2

[Georg Fritzsche [:gfritzsche]]

Javauns metrics wishlist could also be a good starting point for the format:
https://docs.google.com/spreadsheets/d/1gDQbcKkMC1qj75DRsN5t5IkRs5oWMUxRYUnXflybJWw/

Comment 3

[Alessio Placitelli [:Dexter]]

I've started putting things together: https://docs.google.com/spreadsheets/d/1T9_hUxQGg1OwN8UruqIlQKTePUS3BMnKlDctmYbhUbo/edit#gid=0

Comment 4

[Alessio Placitelli [:Dexter]]

Marco, does the format of the document from comment 3 look reasonable to you?

Comment 5

[Marco Bonardo [:mak]]

Does "Enter" also imply click? And well, the same is valid for most of the other entries. Maybe we should split each entry by keyboard/mouse, I'm not sure we have the same test coverage for the 2 interactions.
And also for telemetry, I don't think the current histograms distinguish keyboard and mouse, something UITelemetry does instead and it could be useful from a UX point of view. We may count that as a miss. But regardless, we should likely track status for both separately.

why does it state "open the result in the current tab", it can actually also open in a different tab or window.

re: System search, I'm not sure if it's there to track Cortana searches, but afaict MS removed any possibilities to do those.
Maybe you could replace that with activity stream, not sure.

I don't know if we want already to introduce future possible splittings, like search suggestions may be local or remote (in searchbar, home, newtab), we may want to track those separately.

I don't see anything about the search settings button that is part of one-off experience, do we plan to track it anywhere? I think currently it's only tracked by UITelemetry.

Apart from that, the tabular setup looks ok

Comment 6

[Alessio Placitelli [:Dexter]]

(In reply to Marco Bonardo [::mak] from comment #5)
> Does "Enter" also imply click? And well, the same is valid for most of the
> other entries. Maybe we should split each entry by keyboard/mouse, I'm not
> sure we have the same test coverage for the 2 interactions.
> And also for telemetry, I don't think the current histograms distinguish
> keyboard and mouse, something UITelemetry does instead and it could be
> useful from a UX point of view. We may count that as a miss. But regardless,
> we should likely track status for both separately.

Yes, that's basically indicating the "simple search" you can perform by writing in the awesomebar/seachbar and then simply pressing enter or clicking.

The current histograms/scalars don't make any distinction on the triggering input device. Feel free to add/edit the document as you like!

> why does it state "open the result in the current tab", it can actually also
> open in a different tab or window.

Whops, by bad, removed that.

> re: System search, I'm not sure if it's there to track Cortana searches, but
> afaict MS removed any possibilities to do those.
> Maybe you could replace that with activity stream, not sure.

Mh, that's for tracking [1], on which I've accidentally stumbled upon and I'm *assuming* it's for system search/Cortana. Unfortunately, I have no clue. Since the code is still there, I'll leave it on the spreadsheet with a note. We can always remove the entry if we remove the code.

> I don't know if we want already to introduce future possible splittings,
> like search suggestions may be local or remote (in searchbar, home, newtab),
> we may want to track those separately.
> 
> I don't see anything about the search settings button that is part of
> one-off experience, do we plan to track it anywhere? I think currently it's
> only tracked by UITelemetry.

I agree this should probably be covered by QA efforts, as other things that I'm probably not capturing in the spreadsheet because I don't know about them.

Once I get this structure going, I think the best path forward would be for you or :past (or somebody with a greater knowledge of the search area) to help filling in the gaps.

[1] - https://dxr.mozilla.org/mozilla-central/rev/8103c612b79c2587ea4ca1b0a9f9f82db4b185b8/browser/components/nsBrowserContentHandler.js#246

Comment 7

[Alessio Placitelli [:Dexter]]

Panos, we filled the spreadsheet from comment 3! Can you take it from here?

Comment 8

[Panos Astithas (he/him) [:past] (please ni?)]

Yes, thank you for putting this together!

Comment 9

[Alessio Placitelli [:Dexter]]

(In reply to Panos Astithas [:past] from comment #8)
> Yes, thank you for putting this together!

My pleasure!

Comment 10

[Panos Astithas (he/him) [:past] (please ni?)]

I filed a number of implementation bugs blocking bug 1334599, so I think the work this bug was supposed to track is now completed. Feel free to reopen if I misunderstood.

Comment 11

[Georg Fritzsche [:gfritzsche]]

Are the gaps in test coverage filled now or do we have bugs filed for them?
Are the open questions in the document resolved or planned to be addressed in a separate bug?
Is there a plan to have QA work through the expected behavior in the doc to make sure the search features behave as intended?

Comment 12

[Panos Astithas (he/him) [:past] (please ni?)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> Are the gaps in test coverage filled now or do we have bugs filed for them?

The implementation bugs blocking bug 1334599 will come with automated tests. Are there any other gaps to fill?

> Are the open questions in the document resolved or planned to be addressed
> in a separate bug?

It's not clear which document you are referring to, but I tried to distill both spreadsheets (Javaun's and Alessio's) to implementation bugs. So I believe I incorporated every question in the spreadsheet comments.

> Is there a plan to have QA work through the expected behavior in the doc to
> make sure the search features behave as intended?

Yes, the plan is to have QA formulate a test plan based on both spreadsheets and bugs on file, once we have made more progress into the implementation.

Comment 13

[Georg Fritzsche [:gfritzsche]]

(In reply to Panos Astithas [:past] from comment #12)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> > Are the gaps in test coverage filled now or do we have bugs filed for them?
> 
> The implementation bugs blocking bug 1334599 will come with automated tests.
> Are there any other gaps to fill?

Alessio?

Comment 14

[Alessio Placitelli [:Dexter]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> (In reply to Panos Astithas [:past] from comment #12)
> > (In reply to Georg Fritzsche [:gfritzsche] from comment #11)
> > > Are the gaps in test coverage filled now or do we have bugs filed for them?
> > 
> > The implementation bugs blocking bug 1334599 will come with automated tests.
> > Are there any other gaps to fill?
> 
> Alessio?

Mh, I can't directly spot the test coverage for the in-content searches (i.e. the about:* sections in [1]). If that's covered by other bugs there, we should be ok (at least, considering what I wrote in [1]).

[1] - https://docs.google.com/spreadsheets/d/1T9_hUxQGg1OwN8UruqIlQKTePUS3BMnKlDctmYbhUbo/edit#gid=2096837827

Comment 15

[Panos Astithas (he/him) [:past] (please ni?)]

Alessio you are right, I haven't filed bugs for about:home and about:newtab yet. We consider them part of the second phase of the project and wanted to complete phase one first. This is still in my TODO list, but I'll reopen this bug as an additional reminder and opportunity to provide updates.

Comment 16

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.

Comment 17

[Mark Banner (:standard8)]

Panos, was there still more to do here based on comment 15, or did that get completed eventually?

Comment 18

[Panos Astithas (he/him) [:past] (please ni?)]

I don't remember filing those bugs and I don't know how our plans evolved here.

Comment 19

[Mark Banner (:standard8)]

Note for reference, comment 14 has a useful spreadsheet with the summary.

Comment 20

[Mark Banner (:standard8)]

Having recently gone through most of the probes and the tests for some other work we did on re-arranging the logging, I'm not pretty confident we have the coverage we need. We also have a lot simpler code.

I looked through some of the missing areas referenced in the spreadsheet in comment 14 and I think we're covering all of those now.