1459947 - Various webextension tests are going to permafail when the Gecko version number is bumped to 63 on 2018-06-25

Status: VERIFIED FIXED

(WebExtensions :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=67f7a605b345437aa7e33d58076cf6b5ac7069ae&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=177472940

toolkit/components/extensions/test/xpcshell/test_ext_background_telemetry.js fails because WEBEXT_BACKGROUND_PAGE_LOAD_MS expires: 
https://treeherder.mozilla.org/logviewer.html#?job_id=177472919&repo=try

toolkit/components/extensions/test/mochitest/test_chrome_ext_contentscript_telemetry.html because of WEBEXT_CONTENT_SCRIPT_INJECTION_MS: https://treeherder.mozilla.org/logviewer.html#?job_id=177473075&repo=try

browser/components/extensions/test/browser/test-oop-extensions/browser_ext_browserAction_telemetry.js because of WEBEXT_BROWSERACTION_POPUP_OPEN_MS: https://treeherder.mozilla.org/logviewer.html#?job_id=177472926&repo=try

Check the list below from the last expiration bump (60 -> 63)-

+++ This bug was initially created as a clone of Bug #1425957 +++

[Tracking Requested - why for this release]: Permafailing tests when the Gecko version number is bumped to 60.

Looks like the probes these tests depend on are set to expire when Gecko hits 60:
WEBEXT_BACKGROUND_PAGE_LOAD_MS
WEBEXT_CONTENT_SCRIPT_INJECTION_MS
WEBEXT_EXTENSION_STARTUP_MS
WEBEXT_STORAGE_LOCAL_GET_MS
WEBEXT_STORAGE_LOCAL_SET_MS

Looks like there's some other expiring probes as well, but those are the ones causing test failures anyway. How do you want to proceed, Bob?

Comment 1

[Andrew Swan [:aswan]]

Bob isn't working on webextensions any more, Luca is handling telemetry now.  I think this may just need to be dup'ed to bug 1452685.

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Andrew Swan [:aswan] from comment #1)
> Bob isn't working on webextensions any more, Luca is handling telemetry now.
> I think this may just need to be dup'ed to bug 1452685.

Yes, all the probes mentioned in comment 0 are the same that bug 1452685's patch is going to re-new.

Comment 3

[Luca Greco [:rpl] [:luca] [:lgreco]]

This should have been fixed by Bug 1452685, I'm closing it as fixed (but let me know / re-open it if on the next "version increase simulation" this issue is still reproducible).

Comment 4

[Andreea Pavel [:apavel]]

These failures no longer appear on recent beta simulations.

Comment 5

[Intermittent Failures Robot]

14 failures in 669 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 14

Platform breakdown:
* linux32-devedition: 4
* windows10-64-devedition: 2
* linux64: 2
* windows10-64: 1
* osx-10-10: 1
* macosx64-devedition: 1
* linux64-nightly: 1
* linux32-nightly: 1
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1459947&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Andreea Pavel [:apavel] from comment #4)
> These failures no longer appear on recent beta simulations.
Has this been verified with a version increase simulation?

Comment 7

[Andreea Pavel [:apavel]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #6)
> (In reply to Andreea Pavel [:apavel] from comment #4)
> > These failures no longer appear on recent beta simulations.
> Has this been verified with a version increase simulation?

No, just by checking the latest 3/4 beta simulations. 
Can we take a look at it today?

Comment 8

[Andreea Pavel [:apavel]]

Successful version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6d13a67716f83a9fd7bead7acc57daf6a5d4460e&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified