1389994 - Various browser_UsageTelemetry tests are going to permafail when the Gecko version number is bumped to 58 on 2017-09-18

Status: VERIFIED FIXED

(Firefox :: Search, defect, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

[Tracking Requested - why for this release]: Multiple browser-chrome test failures when the Gecko version number gets bumped to 58

Not entirely sure I understand this since the scalars in question all appear to have an expiration date of never. That said, the failures are clearly happening. Alessio, can you please take a look? :)

https://treeherder.mozilla.org/logviewer.html#?job_id=122804190&repo=try

TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_UsageTelemetry_content.js | Should have matching amount of events. - 0 == 1 - JS frame :: chrome://mochitests/content/browser/browser/modules/test/browser/head.js :: checkEvents :: line 113
TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_UsageTelemetry_searchbar.js | Should have matching amount of events. - 0 == 1 - JS frame :: chrome://mochitests/content/browser/browser/modules/test/browser/head.js :: checkEvents :: line 113
TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_UsageTelemetry_urlbar.js | Should have matching amount of events. - 0 == 1 - JS frame :: chrome://mochitests/content/browser/browser/modules/test/browser/head.js :: checkEvents :: line 113

Comment 1

[Alessio Placitelli [:Dexter]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #0)
> [Tracking Requested - why for this release]: Multiple browser-chrome test
> failures when the Gecko version number gets bumped to 58
> 
> Not entirely sure I understand this since the scalars in question all appear
> to have an expiration date of never. That said, the failures are clearly
> happening. Alessio, can you please take a look? :)

This bug was a nice warm welcome back from PTO :-P

So, this seems to have nothing to do with scalars: all the failures in the log mention "events", and the search events seem to be expiring on 58 [1], so the problems look legit. Since this doesn't seem to be a problem with the collection mechanism, I'm moving it to Firefox->Search.

@past/@dzeber - you are listed as the owners of the probes. Are you planning on extending the collection? If not, then the test should be cleaned up otherwise they will keep failing on 58.

[1] - http://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#15

Comment 2

[Panos Astithas (he/him) [:past] (please ni?)]

I believe we do want them, we didn't get a chance to use them a lot so far.

Comment 3

[Dave Zeber [:dzeber]]

Yes, we do plan to use them in future studies, but there is some setup time required for pipelining/analysis that we haven't had a chance to work on yet given the 57 crunch.

Comment 4

[Alessio Placitelli [:Dexter]]

Please extend the collection period in the probe definition and ask for a new data review ;-)

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Attachment: extend the expiration to 65

Extending the probe expiration to 65 gets us to mid-October of next year. I assume that's long enough for everyone without being an outright never?

Comment 6

[Panos Astithas (he/him) [:past] (please ni?)]

Comment on attachment 8900020 [details] [diff] [review]
extend the expiration to 65

Review of attachment 8900020 [details] [diff] [review]:
-----------------------------------------------------------------

Sounds good to me, thanks!

Comment 7

[[e-mail]]

Comment on attachment 8900020 [details] [diff] [review]
extend the expiration to 65

Review of attachment 8900020 [details] [diff] [review]:
-----------------------------------------------------------------

This is an extension of existing probes.  Fine to extend.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ba609cde83c6
Extend the search events probe expiration to Firefox 65. r=rweiss

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/ba609cde83c6