Closed Bug 1464990 Opened 6 years ago Closed 6 years ago

Allow easier testing Switchboard experiments

Categories

(Firefox for Android Graveyard :: Metrics, enhancement)

All
Android
enhancement
Not set
normal

Tracking

(firefox62 verified)

VERIFIED FIXED
Firefox 62
Tracking Status
firefox62 --- verified

People

(Reporter: petru, Assigned: petru)

Details

(Whiteboard: [priority:medium] )

Attachments

(1 file)

Currently to be able to test certain features depending on Switchboard experiments on nightly builds, the QA team has to
- first install a build older than v56, 
- enable the desired experiments with the help of a Firefox addon specially created by Sebastian [1]
- install the most recent nightly
- test the feature

This is a frail process and as seen in https://bugzilla.mozilla.org/show_bug.cgi?id=1454686#c89 can lead to inconsistencies which makes testing harder.

To address this I see 4 solutions (depending on the current experiments configuration [2] and how that is handled inside the app):
- update the Switchboard Experiments addon to support latest Firefox. The initial addon was created as part of bug 1270929 but is currently incompatible with Quantum. Would be great if someone could continue the work and update it.
- include nightly builds in the list of appIds for which certain experiments are available (some are currently only available on release and beta). This would still allow for certain other attributes to exclude the tested build though (like buckets, regions, etc).
- allow for adb extras to override the availability of a certain experiment. This would use the same functionality the addon uses - store the status of a certain experiment in a preference that would override the normal configuration when checking which experiments should be available [3].
- use a different configuration JSON that would contain the same experiments but with increased availability (no limiting attributes). This functionality would again rely on an intent extra to pass the new URL but this functionality already exists [4].


[1] https://addons.mozilla.org/en-US/android/addon/switchboard-experiments/
[2] https://firefox.settings.services.mozilla.com/v1/buckets/fennec/collections/experiments/records
[3] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/switchboard/SwitchBoard.java#134
[4] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/BrowserApp.java#1015
NIing Sebastian to ask if he has any plans of updating the addon.
Flags: needinfo?(s.kaspari)
Adding NI for Susheel to figure out next steps.
Flags: needinfo?(s.kaspari) → needinfo?(sdaswani)
I spoke to Sebastian about this. He said the following:

if the softvision contractors wanted to resurrect your switchboard extension
to work with fennec
_could_ they (is it possible)?
sebastian [10:27 AM]
Probably not as an extension. But this could be put into Fennec itself where we still have access to the APIs.
susheel [10:28 AM]
ok good to know
could they reuse some of your code?
sebastian [10:30 AM]
Yes, it may be possible to re-use almost all of it and just run it from browser.js in Fennec

Keeping NI to prio
Flags: needinfo?(sdaswani)
Whiteboard: [priority:medium]
Assignee: nobody → petru.lingurar
Status: NEW → ASSIGNED
I think the above patch should be the perfect replacement for Sebastian's addon.

One thing that needs attention: this would allow any user to easily enable / disable Switchboard experiments.
This was possible with the addon also, but if needed it can be restricted to not be available in the release version.
Attachment #8986450 - Flags: review?(sdaswani) → review?(s.kaspari)
Comment on attachment 8986450 [details]
Bug 1464990 - Allow easier testing Switchboard experiments;

https://reviewboard.mozilla.org/r/251812/#review258290

LGTM.

::: mobile/android/components/AboutRedirector.js:73
(Diff revision 1)
>      privileged: true
>    },
> +  experiments: {
> +    uri: "chrome://browser/content/aboutExperiments.xhtml",
> +    privileged: true
> +  },

Does it need to priviledged to use the experiments API? (I do not know).

How about we add "hide: true". I assume that will prevent the page to show about on about:about?
Attachment #8986450 - Flags: review?(s.kaspari) → review+
@Jim: Can you do a second review (or pass it on)? I haven't looked at Fennec code in quite a while and I may miss things.
Comment on attachment 8986450 [details]
Bug 1464990 - Allow easier testing Switchboard experiments;

https://reviewboard.mozilla.org/r/251812/#review258324

Looks good!

::: mobile/android/chrome/content/browser.js:5630
(Diff revision 1)
>  
>      if (aState & Ci.nsIWebProgressListener.STATE_IS_SECURE) {
>        return this.IDENTITY_MODE_IDENTIFIED;
>      }
>  
> -    let whitelist = /^about:(about|accounts|addons|buildconfig|cache|config|crashes|devices|downloads|fennec|firefox|feedback|home|license|logins|logo|memory|mozilla|networking|privatebrowsing|rights|serviceworkers|support|telemetry|webrtc)($|\?)/i;
> +    let whitelist = /^about:(about|accounts|addons|buildconfig|cache|config|crashes|devices|downloads|fennec|firefox|feedback|home|license|logins|logo|memory|mozilla|networking|privatebrowsing|rights|serviceworkers|support|telemetry|webrtc|experiments)($|\?)/i;

Please use alphabetical order.

::: mobile/android/chrome/jar.mn:50
(Diff revision 1)
> +  content/aboutExperiments.xhtml       (content/aboutExperiments.xhtml)
> +  content/aboutExperiments.js          (content/aboutExperiments.js)

Alphabetical order (above "content/aboutLogins.xhtml")
Attachment #8986450 - Flags: review?(nchen) → review+
Comment on attachment 8986450 [details]
Bug 1464990 - Allow easier testing Switchboard experiments;

https://reviewboard.mozilla.org/r/251812/#review258290

> Does it need to priviledged to use the experiments API? (I do not know).
> 
> How about we add "hide: true". I assume that will prevent the page to show about on about:about?

Tried with `privileged: false` and got `ReferenceError: ChromeUtils is not defined`.
Added `hide: true` and checked that indeed it prevents the new page from showing in `about:about`.
Updated the patch with your suggestions, thanks!
Keywords: checkin-needed
Pushed by ebalazs@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/46ea394f0c30
Allow easier testing Switchboard experiments; r=jchen,sebastian
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/46ea394f0c30
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 62
Flags: qe-verify+
Verified on the latest Nightly, everything works as expected.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
Hardware: Unspecified → All
Version: unspecified → Trunk
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.