1281813 - QA for browser engagement phase 1

Status: VERIFIED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Alessio Placitelli [:Dexter]]

We need to perform some quality checks once we have the browser engagement data (as scalars) flowing out of the client.

Comment 1

[Alessio Placitelli [:Dexter]]

Here's the doc I've been putting together: https://docs.google.com/document/d/1Y8IZP85mg_cbaOud30WamjUh6EIhOu-Re3LZ1mWxwLs/edit

Comment 2

[John Dorlus [:Silne30]]

I am attaching a PDF of all of the test cases that we wrote.

Comment 3

[John Dorlus [:Silne30]]

Attachment: Engagement - TestRail.pdf

Comment 4

[Alessio Placitelli [:Dexter]]

Thanks John, these cases look good. Maybe we could have expanded a bit on the total_uri_count probe QA, as it could get a bit tricky. But I'm probably looking at this from a developer point of view.

Additional questions that comes to my mind when looking at your doc (and thinking of that probe):

- Are we really only counting user-initiated URI loads?
- Are loads from iframes counted?
- Are we really restricted to http(s)?

Comment 5

[John Dorlus [:Silne30]]

Reopening to add some cases to cover (In reply to Alessio Placitelli [:Dexter] from comment #4)

Comment 6

[Georg Fritzsche [:gfritzsche]]

- C4312 seems to be covered by C4963 already
- "2.5. browser.engagement.total_uri_count" needs more cases (what Alessio said and a basic "counting URI navigations")
- "2.6. browser.engagement.unique_domains_count" also seems to miss the basic domain counting / increase

A thought for future efforts:
How about we start out with this kind of list, before committing to the detail work of testing all the cases?

Comment 7

[John Dorlus [:Silne30]]

Sounds like a plan.

Comment 8

[John Dorlus [:Silne30]]

While validating if total_uri_count performs correctly, I noticed a small difference in an edge case. Can you tell me what behavior is desired. 

Steps to reproduce:
1.) Open up nightly with new profile
2.) In the first tab, navigate to about:telemetry and expand the Scalars section. They should read: 
Name: browser.engagement.total_uri_count, Value: 1
Name: browser.engagement.max_concurrent_window_count, Value:1
Name: browser.engagement.unique_domains_count, Value: 1
Name: browser.engagement.max_concurrent_tab_count, Value: 2

3.) Go to the other tab and navigate to about:nobody (It's a fake page, should throw error)
4.) Go back to about:telemetry Tab and refresh. The scalars should not change.

Comment 9

[John Dorlus [:Silne30]]

Steps to reproduce (scenario 2):

Steps to reproduce:
1.) Open up nightly with new profile
2.) In the first tab: navigate to about:nobody
3.) In the second tab: navigate to about:telemetry and expand the scalars section.
4.) The scalars are different from the above scenario. They now read:
Name: browser.engagement.max_concurrent_window_count, Value:1
Name: browser.engagement.max_concurrent_tab_count, Value: 2

Are both scenarios correct? Should changing the order of the operations have yielded these different results? Let me know so I know wha to put in Test Rail.

Comment 10

[John Dorlus [:Silne30]]

(In reply to Alessio Placitelli [:Dexter] from comment #4)
> Thanks John, these cases look good. Maybe we could have expanded a bit on
> the total_uri_count probe QA, as it could get a bit tricky. But I'm probably
> looking at this from a developer point of view.
> 
> Additional questions that comes to my mind when looking at your doc (and
> thinking of that probe):
> 
> - Are we really only counting user-initiated URI loads?
> - Are loads from iframes counted?
> - Are we really restricted to http(s)?

About the first point that you mentioned about only counting user-initiated URI Loads: Would that count the homepage that is loaded when you start nightly? That page is counted as a unique domain yet the user did not initiate it.

Comment 11

[Alessio Placitelli [:Dexter]]

(In reply to John Dorlus [:Silne30] from comment #10)
> (In reply to Alessio Placitelli [:Dexter] from comment #4)
> > - Are we really only counting user-initiated URI loads?
> 
> About the first point that you mentioned about only counting user-initiated
> URI Loads: Would that count the homepage that is loaded when you start
> nightly? That page is counted as a unique domain yet the user did not
> initiate it.

This sounds like a bug. Could you please file one?

Comment 12

[Alessio Placitelli [:Dexter]]

(In reply to John Dorlus [:Silne30] from comment #8)
> While validating if total_uri_count performs correctly, I noticed a small
> difference in an edge case. Can you tell me what behavior is desired. 
> 
> Steps to reproduce:
> 1.) Open up nightly with new profile
> 2.) In the first tab, navigate to about:telemetry and expand the Scalars
> section. They should read: 
> Name: browser.engagement.total_uri_count, Value: 1
> Name: browser.engagement.max_concurrent_window_count, Value:1
> Name: browser.engagement.unique_domains_count, Value: 1
> Name: browser.engagement.max_concurrent_tab_count, Value: 2
> 
> 3.) Go to the other tab and navigate to about:nobody (It's a fake page,
> should throw error)
> 4.) Go back to about:telemetry Tab and refresh. The scalars should not
> change.

I'm not sure about that. Could you please file a bug about this and the other issue that you find and make them block this bug?

Comment 13

[John Dorlus [:Silne30]]

(In reply to Alessio Placitelli [:Dexter] from comment #4)
> Thanks John, these cases look good. Maybe we could have expanded a bit on
> the total_uri_count probe QA, as it could get a bit tricky. But I'm probably
> looking at this from a developer point of view.
> 
> Additional questions that comes to my mind when looking at your doc (and
> thinking of that probe):
> 
> - Are we really only counting user-initiated URI loads?
> - Are loads from iframes counted?
> - Are we really restricted to http(s)?

For the third point, I tried creating a test case that would access an ftp:// site but I cannot find a site to navigate to. Even ftp://ftp.mozilla.org has been changed over to https. Is there any way to get this working or is there another protocol I can use to test this case?

Comment 14

[Alessio Placitelli [:Dexter]]

(In reply to John Dorlus [:Silne30] from comment #13)
> > - Are we really restricted to http(s)?
> 
> For the third point, I tried creating a test case that would access an
> ftp:// site but I cannot find a site to navigate to. Even
> ftp://ftp.mozilla.org has been changed over to https. Is there any way to
> get this working or is there another protocol I can use to test this case?

You can use a data URI, such as:

data:text/html,<html><body id="some_id"><input></body></html>

Comment 15

[John Dorlus [:Silne30]]

Attachment: Engagement - Updated - TestRail.pdf

Comment 16

[Alessio Placitelli [:Dexter]]

(In reply to John Dorlus [:Silne30] from comment #15)
> Created attachment 8792124 [details]
> Engagement - Updated - TestRail.pdf

Thanks John! I can't seem to find any test covering basic URI counting (e.g. open a list of URIs and making sure that the number of URIs after a browsing session matches). Is that included in some other test?

Comment 17

[John Dorlus [:Silne30]]

Attachment: Engagement - Second Update - TestRail.pdf

Comment 18

[John Dorlus [:Silne30]]

Comment on attachment 8793360 [details]
Engagement - Second Update - TestRail.pdf

Added tests cases to cover both basic cases.

Comment 19

[Alessio Placitelli [:Dexter]]

Comment on attachment 8793360 [details]
Engagement - Second Update - TestRail.pdf

This looks good to me after your changes.

Comment 20

[Georg Fritzsche [:gfritzsche]]

I think Alessio has this covered now, thanks!