1525391 - Collect telemetry for request time and cache age for Pocket

Status: VERIFIED FIXED

(Firefox :: New Tab Page, defect, P1)

Description

[Kate Hudson :k88hudson]

Due to Product's feedback that performance and content freshness will have an impact on experiment success, we would like to capture the following information:

• Request time for layout endpoint
• Request time for feed endpoints
• Request time for spoc endpoint
• Total request time for data completeness / time to first render (do we already have this?)
• Cache age on browser start-up

Comment 1

[Nan Jiang [:nanj]]

k88hudson - are we only interested in the first round of endpoint requests on browser start-up, or do we also want to collect the request time for all the subsequent requests?

Comment 2

[Kate Hudson :k88hudson]

I think we only care about the first round after the start-up of the browser.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4766

Comment 4

[Nan Jiang [:nanj]]

Attachment: data_review_request_pockets_performance.txt

Hey Kenny, could you do a data review for us? See the attachment for the data review request form.

Thanks!

Comment 5

[Nan Jiang [:nanj]]

Attachment: data_review_request_pockets_performance.txt

Comment 6

[klong]

Attachment: data_review_newtab_pocket_performance.txt

r+

Comment 7

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/6d9b468951ce

Comment 8

[Kate Hudson :k88hudson]

Nan, can you add steps to verify to this bug and needinfo brahmini when you've posted them? Thanks

Comment 9

[Nan Jiang [:nanj]]

I've updated the QA document with details about the test plan.

Brahmini, you can find it in the "Performance pings" section of the "QA plan for Pocket + New Tab Experiment".

Comment 10

[Brahmini Nagabandi]

Hi, I can find it in QA Plan https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.3zj0kjmo4xno.Thanks.

Comment 11

[Brahmini Nagabandi]

Attachment: QA - bug 1525391.png

Following steps as per QA plan for Pocket + New Tab Experiment doc

I used 2 different end-points for pref browser.newtabpage.activity-stream.discoverystream.config :

• {"enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all1"} and
• {"enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all2"}

Cache age request is missing. (can be found in attachment)

Attachment QA - bug 1525391.png

Please let me know if I'm missing anything, thanks.

Comment 12

[Nan Jiang [:nanj]]

Hi Brahmini,

Thanks for pointing this out! The cache age will only be reported if there exists a cache file. There are two cases where the cache file won't exist:

• When you create a Firefox profile
• When you change anything of the discovery stream pref, which will delete the old cache file

I will put this to the QA document.

BTW, looks like both endpoints in comment 11 are broken in the nightly. I am getting these errors:

Failed to fetch https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all1: https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all1 returned unexpected status: 400 DiscoveryStreamFeed.jsm:97
No response for response.layout prop DiscoveryStreamFeed.jsm:166
Failed to fetch https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all1: https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=dev-test-all1 returned unexpected status: 400 DiscoveryStreamFeed.jsm:97
No response for response.layout prop

Comment 13

[Kate Hudson :k88hudson]

Attachment: Bug 1525391 - Collect telemetry for request time and cache age for Pocket

Comment 14

[Kate Hudson :k88hudson]

Ok, can you review the uplift patch and make the uplift form request?

Comment 15

[Tim Spurway [:tspurway]]

Comment on attachment 9044255 [details]
Bug 1525391 - Collect telemetry for request time and cache age for Pocket

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

None

User impact if declined

the product team will be unable to verify the performance of the new tab page in the field, possibly leading to undetected poor performance profile

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

Yes

If yes, steps to reproduce

(see comment #11)

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

this code is an extension to existing telemetry, and any errors raised will not be user facing

String changes made/needed

Comment 16

[Brahmini Nagabandi]

Attachment: QA Results - bug 1525391 (Mac OS).png

QA Results:

Tested on :

FF Nightly version : 67.0a1 (2019-02-15)
OS : Mac and Windows 10 Pro

Results attached QA Results - bug 1525391 (Mac OS).png

Works as expected.

Marking as verified.

Comment 17

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9044255 [details]
Bug 1525391 - Collect telemetry for request time and cache age for Pocket

Planned work for pocket/new tab. Verified in Nightly.
OK for beta uplift, should land for beta 9.
Landing order: bug 1519879, bug 1525494, bug 1526861, bug 1524669, bug 1527195, bug 1525391, bug 1527347, bug 1525366, bug 1527626, bug 1527397, bug 1518258, bug 1527701, bug 1527370.

Comment 18

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/86bd7fce23cd

Comment 19

[Camelia Badau [:cbadau], Desktop QA]

(In reply to Brahmini Nagabandi from comment #16)

Created attachment 9044324 [details]
QA Results - bug 1525391 (Mac OS).png

QA Results:

Tested on :

FF Nightly version : 67.0a1 (2019-02-15)
OS : Mac and Windows 10 Pro

Results attached QA Results - bug 1525391 (Mac OS).png

Works as expected.

Marking as verified.

Can you please verify this issue on Firefox 66 Beta 9 (https://archive.mozilla.org/pub/firefox/candidates/66.0b9-candidates/build1/)?

Comment 20

[Brahmini Nagabandi]

BETA Testing:

QA Results:

Tested on :

FF Beta version : 66.0b9 (64-bit)
OS : Mac and Windows 10 Pro
Date : Feb 22

End-point used : {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-beta-2-complex"}

Verified on the latest Beta. Works as expected.

Attaching QA results below for both Mac and Windows

Marking as verified.

Comment 21

[Brahmini Nagabandi]

Attachment: QA Results:BETA - 1525391 (Mac OS).png

Comment 22

[Brahmini Nagabandi]

Attachment: QA Results:BETA - 1525391 (Windows OS).png

Comment 23

[Camelia Badau [:cbadau], Desktop QA]

Thanks for verifying this!