1471204 - browser/modules/test/browser/browser_BrowserErrorReporter.js is going to permafail when the Gecko version number is bumped to 64 on 2018-09-04

Status: VERIFIED FIXED

(Firefox :: General, defect)

Description

[Narcis Beleuzu [:NarcisB]]

Version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=26c0f189327c7888e1505766ca78be45f559f990&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=runnable&selectedJob=184890069

Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=184890069&repo=try&lineNumber=3166

Log info:
[task 2018-06-26T11:02:11.140Z] 11:02:11     INFO - TEST-START | browser/modules/test/browser/browser_BrowserErrorReporter.js
[task 2018-06-26T11:02:11.566Z] 11:02:11     INFO - TEST-INFO | started process screentopng
[task 2018-06-26T11:02:12.353Z] 11:02:12     INFO - TEST-INFO | screentopng: exit 0
[task 2018-06-26T11:02:12.353Z] 11:02:12     INFO - Buffered messages logged at 11:02:11
[task 2018-06-26T11:02:12.353Z] 11:02:12     INFO - Entering test bound testSetup
[task 2018-06-26T11:02:12.353Z] 11:02:12     INFO - Leaving test bound testSetup
[task 2018-06-26T11:02:12.354Z] 11:02:12     INFO - Entering test bound testInitPrefDisabled
[task 2018-06-26T11:02:12.354Z] 11:02:12     INFO - TEST-PASS | browser/modules/test/browser/browser_BrowserErrorReporter.js | Reporter does not listen for errors if the enabled pref is false. - 
[task 2018-06-26T11:02:12.356Z] 11:02:12     INFO - Leaving test bound testInitPrefDisabled
[task 2018-06-26T11:02:12.357Z] 11:02:12     INFO - Entering test bound testInitUninitPrefEnabled
[task 2018-06-26T11:02:12.359Z] 11:02:12     INFO - Console message: [JavaScript Warning: "Use of nsIFile in content process is deprecated." {file: "resource://gre/modules/FileUtils.jsm" line: 170}]
[task 2018-06-26T11:02:12.360Z] 11:02:12     INFO - TEST-PASS | browser/modules/test/browser/browser_BrowserErrorReporter.js | Reporter listens for errors if the enabled pref is true. - 
[task 2018-06-26T11:02:12.362Z] 11:02:12     INFO - TEST-PASS | browser/modules/test/browser/browser_BrowserErrorReporter.js | Reporter does not listen for errors after uninit. - 
[task 2018-06-26T11:02:12.363Z] 11:02:12     INFO - Leaving test bound testInitUninitPrefEnabled
[task 2018-06-26T11:02:12.364Z] 11:02:12     INFO - Entering test bound testEnabledPrefWatcher
[task 2018-06-26T11:02:12.368Z] 11:02:12     INFO - TEST-PASS | browser/modules/test/browser/browser_BrowserErrorReporter.js | Reporter does not collect errors if the enable pref is false. - 
[task 2018-06-26T11:02:12.370Z] 11:02:12     INFO - Console message: [JavaScript Error: "Shouldn't report"]
[task 2018-06-26T11:02:12.374Z] 11:02:12     INFO - TEST-PASS | browser/modules/test/browser/browser_BrowserErrorReporter.js | Reporter collects errors if the enabled pref switches to true. - 
[task 2018-06-26T11:02:12.376Z] 11:02:12     INFO - Leaving test bound testEnabledPrefWatcher
[task 2018-06-26T11:02:12.377Z] 11:02:12     INFO - Entering test bound testScalars
[task 2018-06-26T11:02:12.378Z] 11:02:12     INFO - Buffered messages finished
[task 2018-06-26T11:02:12.379Z] 11:02:12     INFO - TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_BrowserErrorReporter.js | browser.errors.collected_count is incremented when an error is collected. - Got undefined, expected 7

Comment 1

[Osmose [:osmose, :mkelly]]

It looks like telemetry isn't being collected at all during the test, hence the "undefined". MOZ_DATA_REPORTING should be true, and we have a head file pulled in during the test[1] that sets Services.telemetry.canRecordExtended to true[2].

mreid: Do you know of any reasons why telemetry would return no data for this try run?

[1] https://searchfox.org/mozilla-central/source/browser/modules/test/browser/browser_BrowserErrorReporter.js#13
[2] https://searchfox.org/mozilla-central/rev/14cb8f1c238735ba1abe18ad44e39808983c2572/browser/modules/test/browser/head_BrowserErrorReporter.js#23

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The probe browser.errors.collected_count expires when Gecko hits version 64: https://dxr.mozilla.org/mozilla-central/rev/681eb7dfa324dd50403c382888929ea8b8b11b00/toolkit/components/telemetry/Scalars.yaml#1649

Comment 3

[Osmose [:osmose, :mkelly]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=61d4aa0e1eaa12295bc26bdffeed6d8682a49515

Comment 4

[Osmose [:osmose, :mkelly]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5e21200e1fae71446ad1ba92fd97b204a88df53f

Comment 5

[Osmose [:osmose, :mkelly]]

Attachment: Bug 1471204: Disable browser error telemetry tests when scalars expire.

Comment 6

[Phabricator Automation]

Comment on attachment 8988920 [details]
Bug 1471204: Disable browser error telemetry tests when scalars expire.

Chris H-C :chutten has approved the revision.

https://phabricator.services.mozilla.com/D1891

Comment 7

[Phabricator Automation]

Comment on attachment 8988920 [details]
Bug 1471204: Disable browser error telemetry tests when scalars expire.

:Gijs (he/him) has approved the revision.

https://phabricator.services.mozilla.com/D1891

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86416094aaf1
Disable browser error telemetry tests when scalars expire. r=Gijs,chutten

Comment 9

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/86416094aaf1

Comment 10

[Andreea Pavel [:apavel]]

No longer appears on latest beta simulation https://treeherder.mozilla.org/#/jobs?repo=try&revision=f650af024c9f92eae8e7ce2d7a407a4d306b36f0&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=189406070

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Andreea Pavel [:apavel] from comment #10)
> No longer appears on latest beta simulation
This only fails in version increase simulations.

Comment 12

[Andreea Pavel [:apavel]]

Sorry about that :(

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed in yesterday's 63-as-64 simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cad7786268171af69bfad6462aa40587a2a03c0a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable