1432236 - [Acquisition] a/b test in /new var w. Developer edition

Status: RESOLVED FIXED

(Websites :: Web Analytics, task)

Description

[Fabio Rios [:frios]]

We believe there are developers downloading the release version of Firefox every day because they don't know that there is a developer edition available. While people using the release version of Firefox can still access developer tools, they miss out on additional tools and benefits that are unique to Firefox Developer Edition. We will create a test that offers Developer Edition as a secondary offer on our download page in order properly route users to the best version of Firefox for them.

Additional experiment detail here: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=72778147

Optimizely test here: https://app.optimizely.com/edit?experiment_id=10152850018

Comment 1

[Fabio Rios [:frios]]

@Hoosteno can you help me confirm a few details?

- is tracking for clicks to the dev edition link/button in variations B and C set up?
- will we be able to measure conversions if someone clicks the dev edition button and then downloads dev edition?
- I set it up to limit this experiment to en-US, US, desktop, referral traffic. Is that what you see?

Anything else? I have very limited html knowledge. We'll need web team to verify the new button and links are not disrupting anything else on the page.

Comment 2

[Alex Gibson [:agibson]]

Some thoughts on this experiment from a web dev perspective:

1.) The hypothesis that web developers should be using Dev Edition over release channel is not quite black & white. Web developers *need* to test their websites in the release version of Firefox, as it's the same version that the general public (and their clients use). I would expect every web developer worth their salt to want to download the main release version (as well as Developer Edition). I think the correct place to push a link for Dev Edition would be after someone has downloaded Firefox, so as not to hurt conversions. This could be on /firefox/new/?scene=2, or even better after the first time someone opens Developer Tools in release channel (would require some in-product work).

2.) This experiment does little to explain what Dev Edition actually is, or why someone should use it. This just being an experiment aside, placing buttons like this on the main download page would only likely detract from the main CTA, and also place some of our general audience unknowingly in the Beta channel. I think this is unlikely to help retention and not something we would ever want do on this page in this fashion.

3.) We already have a landing page dedicated for Dev Edition [1]. If this experiment is targeting a particular audience demographic, this target destination would provide much more context as to what Dev Edition is, and why web developers should choose it.

4.) If we're worried that we're losing audience for Dev Edition and people are downloading the main release version instead, we should be thinking about promoting the Dev Edition download page more prominently on the website. The page is now burried on mozorg (save for a small link in the footer), as it's no longer found in our main navigation.

[1] https://www.mozilla.org/en-US/firefox/developer/

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

I agree with all the concerns in comment 2 and raised them a week ago. I hesitate to build an acquisition experiment that we'd never put into production (which I believe is true of this one). Fabio convinced me this was a worthwhile experiment by reframing it. I'd like to urge Fabio to put this reframing into the experiment description. Here's how I understand it:

This isn't a test to learn if we should put a dev edition download button on /new for all visitors. This test will teach us 2 things that we will be glad to know going forward:
1) Are there any sources of referral that make sense to show a developer CTA instead of or along side the GA CTA? Answering this question will require us to instrument a way to track the referrer. Since this is proposed as a Traffic Cop experiment, can we expect that TC will filter by "referred" and pass through any/all referrer information it gets to GA? It'll have to for #1 to be true.
2) Does having a second CTA (of any sort) have an impact? Do people click it? Does it reduce clicks on the primary CTA?

With that framing (and assuming we can measure those things), I support running this experiment for a limited time on a subset of visitors. I recommend the subset of visitors should include _all sources of referral that ended in a DevEdition download in 2017_.

Fabio, if we're concerned about volume of Dev Edition downloads, I agree with comment 2 point 1 that we should re-establish the doorhanger in Release devtools[0]. Anyone in release should know about dev edition.

[0] https://bug1374961.bmoattachments.org/attachment.cgi?id=8879862 was created in https://bugzilla.mozilla.org/show_bug.cgi?id=1374961

Comment 4

[Stephanie Hobson [:shobson]]

I have confirmed with the team we have the technical ability to do this :) If the user has configured their browser to keep the referrer private we will not be able to get the referrer. But, if they are sharing that information we can act on it.

Comment 5

[Fabio Rios [:frios]]

Hey all, thanks for thinking this through with me. We're making this experiment better.

I spent some time in GA and it's going to be too difficult to hone in on sources that are "likely" more developer focused. 

I'd like to propose that we try this backwards. We run this to all referral traffic, then see which sources (if any) index higher for developers. I know it feels a bit like we'll reach too many non-devs here, but given the download rate from referral sources is already low, we can experiment a bit and potentially learn something that will lead us to a better experience. Have a look at the updated mana. I think this addresses your points and questions. 

Help me make better here: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=72778147 

(In reply to Alex Gibson [:agibson] from comment #2)
> Some thoughts on this experiment from a web dev perspective:
> 
> ...I think the correct place to push a link for Dev
> Edition would be after someone has downloaded Firefox, so as not to hurt
> conversions. This could be on /firefox/new/?scene=2, or even better after
> the first time someone opens Developer Tools in release channel (would
> require some in-product work).
Alex, I really like this idea. It's going to be a wonky experience since you've just downloaded and we're asking you to download something else. But that's OK. This is a small test to tell us if there is any interest. We'll work on optimizing the experience after we see some success indicators. I've added this as a variation in the test.

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #3)
> With that framing (and assuming we can measure those things), I support
> running this experiment for a limited time on a subset of visitors. I
> recommend the subset of visitors should include _all sources of referral
> that ended in a DevEdition download in 2017_.
I looked at this audience and it's too small. The approach that I'm suggesting now is a bit different but the objectives that we discussed remained the same.

Comment 6

[Alex Gibson [:agibson]]

(In reply to Fabio Rios [:frios] from comment #5)
> Hey all, thanks for thinking this through with me. We're making this
> experiment better.
> 
> I spent some time in GA and it's going to be too difficult to hone in on
> sources that are "likely" more developer focused. 
> 
> I'd like to propose that we try this backwards. We run this to all referral
> traffic, then see which sources (if any) index higher for developers. I know
> it feels a bit like we'll reach too many non-devs here, but given the
> download rate from referral sources is already low, we can experiment a bit
> and potentially learn something that will lead us to a better experience.
> Have a look at the updated mana. I think this addresses your points and
> questions. 
> 
> Help me make better here:
> https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=72778147 
> 
> (In reply to Alex Gibson [:agibson] from comment #2)
> > Some thoughts on this experiment from a web dev perspective:
> > 
> > ...I think the correct place to push a link for Dev
> > Edition would be after someone has downloaded Firefox, so as not to hurt
> > conversions. This could be on /firefox/new/?scene=2, or even better after
> > the first time someone opens Developer Tools in release channel (would
> > require some in-product work).
> Alex, I really like this idea. It's going to be a wonky experience since
> you've just downloaded and we're asking you to download something else. But
> that's OK. This is a small test to tell us if there is any interest. We'll
> work on optimizing the experience after we see some success indicators. I've
> added this as a variation in the test.
> 
> (In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #3)
> > With that framing (and assuming we can measure those things), I support
> > running this experiment for a limited time on a subset of visitors. I
> > recommend the subset of visitors should include _all sources of referral
> > that ended in a DevEdition download in 2017_.
> I looked at this audience and it's too small. The approach that I'm
> suggesting now is a bit different but the objectives that we discussed
> remained the same.

Hypotheses and variations look much better here, thanks. The only area I'd be cautious around from a dev perspective is with the sticky footer. We'd need to make sure it doesn't obscure (or detract from) the primary CTA on smaller screens.

Comment 7

[Fabio Rios [:frios]]

Attachment: dev-call-out.PNG

Hey George, we like this version. Let's roll! Can you create with the updated copy?

Comment 8

[Fabio Rios [:frios]]

Per Krishna's doc: 

Scene 1: (Product first)
Get Firefox Developer Edition to build, test and debug with the latest open-source devtools. Learn more

Scene 2: (Consumer first)
Developers - Looking for open-source devtools? Check out Firefox Developer Edition. 

https://docs.google.com/document/d/1wvWaaCsppdIzHWvWTqah0eNCAV3MlI3xSA3OUCS-vno/edit?ts=5a7ccffe#heading=h.8td3uly30big

Comment 9

[Fabio Rios [:frios]]

Attachment: scene2.PNG

Let's use this version for scene 2

Comment 10

[Fabio Rios [:frios]]

Eric, we're all set.

Comment 11

[George Railean]

Sketch files:

Scene 1: 

https://www.dropbox.com/s/8tkxosm2ozfce6u/Scene_1.sketch?dl=0

Scene 2: 

https://www.dropbox.com/s/ccecftaac082ti2/Scene_2.sketch?dl=0

Thanks,

Comment 12

[Jonathan Petto [:jpetto]]

Question on the mockups:

1. Do we need the Dev Edition callout to be dismissable? This is certainly something we can do and track with a cookie (to ensure it isn't shown on repeated visits), but I want to make sure it's worth the effort.

2. What does the Dev Edition CTA do? Send the visitor to /firefox/developer/? Download Dev Edition directly? (I'm guessing the former, but what to make sure.)

3. How should the Dev Edition callout be displayed on viewports < 1400px wide? Should it continue to be fixed to the bottom right of the existing curved white wave? This may result in the callout being partially (or fully) off-screen in some instances (e.g. 900 x 500px viewport). For viewports < 760px wide, the callout would be rather far from the primary CTA (and may never be seen). We may need some additional mockups here.

Comment 13

[David Tenser [:djst]]

George, I think this design is a little chaotic with new interactions layered on top of a non-interactive product showcase. Have we done internal team review of this - Vince/Jennifer? Want to make sure we're thinking this through.

I also agree with Alex's comments in comment 2 - I'm surprised we want to add this to scene 1 here, I would expect scene 2 to be a more natural flow of the user journey since as a dev you'll want both versions.

Lastly, have we considered adding (back) "Developer Edition" to the sub-nav of /firefox? That's separate from this bug since it's a separate user journey, but it's worth testing if we want to increase exposure to devedition with little risk of reducing consumer downloads.

Comment 14

[Vince Joy]

 Hey David, I'm just learning of this project now. I have similar questions about the design, both with regards to the placement of the new element, and the appropriateness of putting it on scene 1. What do i need to do to get up to speed on how we arrived here?

Comment 15

[Stephanie Hobson [:shobson]]

Let's not lose sight of the goals of the experiment, outlined in comment 3. Putting an additional CTA on this page is the hypothesis that needs testing.

Comment 16

[Eric Renaud [:erenaud]]

I'm washing my hands of this. The test and design meets the Accountable/Approver requirements. 

IMHO we should proceed as is, watch for the results of the test and not waste further time trying make this perfect before shipping.

Comment 17

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #12)
> Question on the mockups:
> 
> 1. Do we need the Dev Edition callout to be dismissable? This is certainly
> something we can do and track with a cookie (to ensure it isn't shown on
> repeated visits), but I want to make sure it's worth the effort.
Either way is fine. If it's more work, I say no since it wouldn't affect the outcome of this test.
> 
> 2. What does the Dev Edition CTA do? Send the visitor to
> /firefox/developer/? Download Dev Edition directly? (I'm guessing the
> former, but what to make sure.)
It links to the /developer/ page. We're simply interested in clicks to the button for this test.
> 
> 3. How should the Dev Edition callout be displayed on viewports < 1400px
> wide? Should it continue to be fixed to the bottom right of the existing
> curved white wave? This may result in the callout being partially (or fully)
> off-screen in some instances (e.g. 900 x 500px viewport). For viewports <
> 760px wide, the callout would be rather far from the primary CTA (and may
> never be seen). We may need some additional mockups here.
Not sure. Would say it should stick above the "Powerfully Private" area.


(In reply to David Tenser [:djst] from comment #13)
> George, I think this design is a little chaotic with new interactions
> layered on top of a non-interactive product showcase. Have we done internal
> team review of this - Vince/Jennifer? Want to make sure we're thinking this
> through.
David, this seemed like an easy way to execute the test. If this proves successful, we could decide to do something entirely different, maybe not even on scene 1. This experiment is not meant to prove that a call out works, but instead that there is an audience we can reach. With that context, good to proceed?

Comment 18

[Eric Renaud [:erenaud]]

David - please see Fabio's comments/questions ^^ comment 17

Comment 19

[David Tenser [:djst]]

Fwiw, I still think the UX is broken but if we believe this will give us valuable data, I'm fine with running it as a test. That said, I have a hard time believing we'd want to keep a CTA on scene on regardless of data results here. I can accept postponing that debate until we have some data though. Thanks!

Comment 20

[Vince Joy]

After reviewing the mockups with Jon Petto, I'm providing some recommendations here for updates to the size and placement of the Dev Edition callout.

1. We shouldn't hide the thing we're trying to test: 
• Move the callout to a z-index above the white curvy shape. 
• Move it away from the right edge of the screen.
• Place it near the rest of the meaningful content on the page.

2. The button should be a link, not a button. 
• Buttons do things, links go places.

3. Shrink the size of the logo .
• At most, the Dev edition logo should be the same size as the Firefox logo at the top of the page. 

Links here:
SlashNew
https://drive.google.com/open?id=1ppyjllBSG2N5auheUrwOMmRY7FYa8lYa

SceneTwo
https://drive.google.com/open?id=1K5pf8ukGm1YfvR_Cfl-G1w4iXh6gp87B

Comment 21

[Jonathan Petto [:jpetto]]

Last thing needed here is the percentage of en-US visitors that should see the experiment. 5%? 10%?

Comment 22

[Jonathan Petto [:jpetto]]

Demo server link:

https://bedrock-demo-jpetto.us-west.moz.works/en-US/firefox/new/

Comment 23

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5467

Comment 24

[Chris More [:cmore]]

I agree with :djst that even if we learn something here, we would unlikely implement the change exactly like this. The design of this experiment appears to have too many competing CTAs and content trying to draw a user's attention. If is possible that the results of this experiment is that we see content splitting down different paths *and* overall volume lowered due to confusion about what is next. Though, that's an assumption and we'll see what the results say. 

Question: how many days at 5% or 10% would we need to determine a conclusion? How quickly will we be able to know if it lowering overall volume? Can we do it at a lower volume and have enough downloads in a reasonable amount of time? Quality downloads are important, but we are still down YoY and total volume still matters.

Comment 25

[pgerman]

5% is good.

Tag management is complete. Expected duration required is 1 day to see if this had a +-5% impact on performance.

@Jpetto: please add the following code after traffic cop has decided which variant will be served 

dataLayer.push({
  'data-ex-present': 'true',
  'data-ex-variant': 'control|dev-link-present',
  'data-ex-experiment': 'fx-dev-ed-on-new'
});

Comment 26

[pgerman]

Report Will be view-able here:https://datastudio.google.com/open/142yb9cq9kNX_ObxVz1X9UKTv5dQXpDBv

Comment 27

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #21)
> Last thing needed here is the percentage of en-US visitors that should see
> the experiment. 5%? 10%?

Hey Jon, confirming this experiment is targeted to en-US, but specifically referral traffic only.

Comment 28

[Fabio Rios [:frios]]

(In reply to Chris More [:cmore] from comment #24)
> Question: how many days at 5% or 10% would we need to determine a
> conclusion? How quickly will we be able to know if it lowering overall
> volume? Can we do it at a lower volume and have enough downloads in a
> reasonable amount of time? Quality downloads are important, but we are still
> down YoY and total volume still matters.

(In reply to pgerman from comment #26)
> Report Will be view-able
> here:https://datastudio.google.com/open/142yb9cq9kNX_ObxVz1X9UKTv5dQXpDBv

I'll keep an eye out for any changes, but let's be real here, this experience isn't disruptive enough to make a notable difference in downloads.

:pegerman, thanks for putting that together! Note that we are not shooting for downloads here. Good to measure, but I'm interested in clicks to the link. The purpose of the experiment is to try and uncover which referral sources are sending us more dev traffic (if any). If I could pick the outcome it would be that traffic from site x and site y are 30% more likely to click the dev CTA and therefore we need to explore how we can leverage traffic from that source better. Be it a partnership with the site, ads on the site, or some specific action on our end. Can we capture that in the dashboard? Also noting we should be targeting referral traffic only in this experiment

Comment 29

[Jonathan Petto [:jpetto]]

(In reply to Fabio Rios [:frios] from comment #27)
> (In reply to Jon Petto [:jpetto] from comment #21)
> > Last thing needed here is the percentage of en-US visitors that should see
> > the experiment. 5%? 10%?
> 
> Hey Jon, confirming this experiment is targeted to en-US, but specifically
> referral traffic only.

Just to clarify, this experiment should only activate for visitors arriving directly to /firefox/new/ from a domain other than mozilla.org. Is that correct?

This experiment will be en-US only.

Comment 30

[pgerman]

(In reply to Fabio Rios [:frios] from comment #28)
> :pegerman, thanks for putting that together! Note that we are not shooting
> for downloads here. Good to measure, but I'm interested in clicks to the
> link. The purpose of the experiment is to try and uncover which referral
> sources are sending us more dev traffic (if any). If I could pick the
> outcome it would be that traffic from site x and site y are 30% more likely
> to click the dev CTA and therefore we need to explore how we can leverage
> traffic from that source better. Be it a partnership with the site, ads on
> the site, or some specific action on our end. Can we capture that in the
> dashboard? Also noting we should be targeting referral traffic only in this
> experiment

Frios: The estimate of one day is dependent on download being the goal of the page, not a link click, we may need it to be live a little bit longer. It will be hard to say how long until we see the volume of clicks the link is getting.

I forwarded you a GA report that shows the top referring sites driving Dev Ed downloads and where they're landing on the site. You might be able to identify some partnerships from that.

Thanks,

PG

Comment 31

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #29)
> (In reply to Fabio Rios [:frios] from comment #27)
> > (In reply to Jon Petto [:jpetto] from comment #21)
> > > Last thing needed here is the percentage of en-US visitors that should see
> > > the experiment. 5%? 10%?
> > 
> > Hey Jon, confirming this experiment is targeted to en-US, but specifically
> > referral traffic only.
> 
> Just to clarify, this experiment should only activate for visitors arriving
> directly to /firefox/new/ from a domain other than mozilla.org. Is that
> correct?
> 
> This experiment will be en-US only.

Yes, that's it.

Comment 32

[Jonathan Petto [:jpetto]]

(In reply to pgerman from comment #25)
> 5% is good.
> 
> Tag management is complete. Expected duration required is 1 day to see if
> this had a +-5% impact on performance.
> 
> @Jpetto: please add the following code after traffic cop has decided which
> variant will be served 
> 
> dataLayer.push({
>   'data-ex-present': 'true',
>   'data-ex-variant': 'control|dev-link-present',
>   'data-ex-experiment': 'fx-dev-ed-on-new'
> });

These are now implemented on the demo server.

:pgerman - Can you verify the tracking info is coming across as expected?

Note that I have upped the chances of getting the variation on the demo server to 80% for testing purposes.

The referrer logic is in place, so you'll need to follow a link from some other site (like bugzilla) to be eligible. (Clicking the link below should do it.)

As always, DNT and tracking protection must be disabled.


https://bedrock-demo-jpetto.us-west.moz.works/en-US/firefox/new/

Comment 33

[pgerman]

Looks good to me. Also, I've created a temporary GA goal for the learn more click.

Look at the updated report: https://datastudio.google.com/reporting/142yb9cq9kNX_ObxVz1X9UKTv5dQXpDBv/page/uGB/edit

Comment 34

[[github robot]]

Commit pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/85daa087384d6bc9751886268e4feea63b6eb981
[fix bug 1432236] Add dev edition experiment to /firefox/new/.

Comment 35

[[github robot]]

Commits pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/114da98769461d965b972ed07b4519f915ee757b
[fix bug 1432236] Enable dev edition experiment on /new.

https://github.com/mozmeao/www-config/commit/4b4fcc2dc8a6ef1d54e0ccc6d24e86dea268e56a
Merge pull request #75 from jpetto/bug-1432236-enable-dev-edition-fx-new-experiment

[fix bug 1432236] Enable dev edition experiment on /new.

Comment 36

[David Tenser [:djst]]

Do we have enough data to make conclusions? Let's stop the test once we have the data we need. Thanks.

Comment 37

[Fabio Rios [:frios]]

(In reply to David Tenser [:djst] from comment #36)
> Do we have enough data to make conclusions? Let's stop the test once we have
> the data we need. Thanks.

Two things have become apparent. 

1) our vanity URLs combined with our own domains make up the bulk of the traffic so that's going to make it difficult to see CTR rates of specific referrers.

- can we exclude our vanity URLs and owned domains from this test?

2) there has been no measurable impact to downloads as a result of this call out. 

- this eases my mined a bit. Can we increase the share of visitors seeing the experiment so that we can increase the clicks? How about an even split

Comment 38

[Eric Renaud [:erenaud]]

Peter - please see/reply to Fabio's questions in comment 37

Comment 39

[pgerman]

This would mean that we would need to update the logic in traffic cop to exclude referrers that belong to a list of our vanity URLs.

jpetto: is that even possible?
Frios: Given that 5/1000 sessions that see 'dev-ed learn more' click it, should we try to learn which sits we should target from comscore/fetch?

Comment 40

[Jonathan Petto [:jpetto]]

Not to pass the buck, but might it be easier to filter out domains on the reporting end? If the test is having no measurable impact on downloads, I'm not sure it matters where the filtering happens.

We can increase the percentage of visitors seeing the variation. Just let me know what we should increase to.

Comment 41

[pgerman]

jpetto: I agree. 
Frios: If you need any help seeing which referrers are generating downloads of dev edition (Goal 5) please let me know.

Comment 42

[Fabio Rios [:frios]]

(In reply to pgerman from comment #39)
> Frios: Given that 5/1000 sessions that see 'dev-ed learn more' click it,
> should we try to learn which sits we should target from comscore/fetch?

Yes, I want to see the CTR from specific sources. (In reply to Jon Petto [:jpetto] from comment #40)
> We can increase the percentage of visitors seeing the variation. Just let me
> know what we should increase to.

Ah good point. Let's increase to 100%. I was going to suggest lower, but I don't see a reason why not to just bump it full and get to the numbers we need faster.

Comment 43

[pgerman]

Frios: Apply this report to www.mozilla.org (production) https://analytics.google.com/analytics/web/template?uid=dzxjnoFQQDyUDXUe4zJ8wg

This should show you which referral sites are sending traffic to our site to download Dev Ed. Maybe I'm missing something, but it feels like ramping up the experiment would have a similar outcome.

Comment 44

[Fabio Rios [:frios]]

(In reply to pgerman from comment #43)
> Frios: Apply this report to www.mozilla.org (production)
> https://analytics.google.com/analytics/web/
> template?uid=dzxjnoFQQDyUDXUe4zJ8wg
> 
> This should show you which referral sites are sending traffic to our site to
> download Dev Ed. Maybe I'm missing something, but it feels like ramping up
> the experiment would have a similar outcome.

At the moment when someone lands on /new, there's no mention of dev edition, so it's very unlikely that they will come across it, unless you're already looking for it at the start. 

This experiment aims to change that by bringing dev messaging forward. The hope is that by doing so, the people that may be interested in dev edition, but don't know about it, will see it when it matters the most -- at the time they are considering downloading Firefox.

I'm just looking for a CTR rate to the dev edition CTA that we've added. I want to know of the people that saw the experiment, how many clicked on the link and which referral source they came from. Unfortunately because our own properties make up the bulk of traffic, I only get a handful of clicks from other referral sources. Ramping up the traffic simply lets the smaller sources have a shot at the experiment.

Comment 45

[pgerman]

Currently we see that 0.5% of folks that land on the page click the Dev Ed. link (Across all referrers). Among the top referrers in the 900-300 sessions range, the CTR varies in the 0-1% range.

Do we expect to find something different if we ramp up the sample?

Comment 46

[Fabio Rios [:frios]]

Maybe not. But we've only got 8000 impressions (referral traffic) to the dev edition notification, so if there is anything else lurking in the results we'll miss it. Seems like little traffic considering the amount of time we've spent on this.

Comment 47

[Eric Renaud [:erenaud]]

David indicated yesterday that we need to shut this off.  Peter's response indicates we've reach statistical significance and any change to the trend would be highly unlikely.

Impetus to close is also the conflict that will arise with the survey test on scene2 to Fx users, which would impact users in the test cohort.

I'll close this bug when that's done.

Comment 48

[[github robot]]

Commits pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/f45a095e026d6748e17a5e72976450bbacfbedce
[fix bug 1432236] Disable Fx new dev edition experiment.

https://github.com/mozmeao/www-config/commit/456610b5b0e8a1823185abe3075d735a97926444
Merge pull request #76 from jpetto/bug-1432236-disable-dev-edition-fx-new-experiment

[fix bug 1432236] Disable Fx new dev edition experiment.

Comment 49

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5493

Comment 50

[[github robot]]

Commit pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/697b5a537d75f45873b302a3777de00dbbc2cdaf
[fix bug 1432236] Remove dev edition experiment files.