Closed Bug 1197271 Opened 10 years ago Closed 10 years ago

test optimized /new against existing /new for improved download conversions

Categories

(www.mozilla.org :: Pages & Content, defect)

Product:
www.mozilla.org
Component:
Pages & Content
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jbertsch, Unassigned)

References

Details

Hypothesis: performance improvements to /new outlined here https://bugzilla.mozilla.org/show_bug.cgi?id=1190614#c0 will increase Firefox download conversions on /new
Set parameters in Optimizely: exclude yahoo source and campaign utm parameters, only desktop users, non Firefox Make sure we don't overlap with any other tests running on /new
John: check out this test and the mentioned above referenced bug in comment 0. We can get this set up together and using our new process to get you up to speed on how to test.
Flags: needinfo?(jkarahalis)
To do: create mana page with test details.
The optimized /new PR [1] has been given an r+. Is there any reason we should wait to merge and push to production? The new URL (/firefox/new/2/) will be unpublished, so no traffic should hit it until the test begins. Are we close to being ready to go? [1] https://github.com/mozilla/bedrock/pull/3232
(In reply to Jon Petto [:jpetto] from comment #4) > The optimized /new PR [1] has been given an r+. Is there any reason we > should wait to merge and push to production? The new URL (/firefox/new/2/) > will be unpublished, so no traffic should hit it until the test begins. Are > we close to being ready to go? > > [1] https://github.com/mozilla/bedrock/pull/3232 I don't see a problem with getting it live before we are ready to do the test. We still need to create the test details and get the optimizely test reviewed.
action for myself: create the mana page details and optimizely test.
Flags: needinfo?(chrismore.bugzilla)
Blocks: 1197395
Hi Cmore- What is the timeframe for getting this live? Thx, Jen
(In reply to Jennifer Bertsch [:jbertsch] from comment #7) > Hi Cmore- > > What is the timeframe for getting this live? > > Thx, > Jen :openjck got the mana page and experiment together today and I have an action to review it all and then we'll move into the code review phase asap. Should get it wrapped up this week to launch on monday.
Flags: needinfo?(jkarahalis)
Flags: needinfo?(chrismore.bugzilla)
Experiment detail: https://app.optimizely.com/edit?experiment_id=3400641306 Optimizely experiment: https://app.optimizely.com/edit?experiment_id=3400641306 When setting up the test, I noticed that webpagetest.org reports that the optimized /new page actually loads more slowly than the original /new page, at least on the first load. That could give us unexpected results. Any idea why that might be happening agibson? It's strange, the pull request looks good to me.
Flags: needinfo?(agibson)
(In reply to John Karahalis [:openjck] from comment #9) > Experiment detail: https://app.optimizely.com/edit?experiment_id=3400641306 > Optimizely experiment: > https://app.optimizely.com/edit?experiment_id=3400641306 > > When setting up the test, I noticed that webpagetest.org reports that the > optimized /new page actually loads more slowly than the original /new page, > at least on the first load. That could give us unexpected results. Any idea > why that might be happening agibson? It's strange, the pull request looks > good to me. When I run the test I see the following results: From: Dulles, VA - Chrome - Cable /firefox/new/ 3.699s, 27 requests (First View) 1.291s 4 requests (Repeated View) /firefox/new/2/ 2.733s, 21 requests (First View) 1.188s, 4 requests (Repeated View) It does seem to vary a bit, depending on repeated runs. I guess network connectivity may also play a role in the results given by webpagetest.org. I can't see how reducing page weight and number of requests could realistically add to the load time, if it weren't for network being in the way :/ Jpetto, any other ideas / thoughts?
Flags: needinfo?(agibson)
Hm. Yeah, the more I run the tests the more the results vary. My earlier results could have been a fluke.
(In reply to John Karahalis [:openjck] from comment #9) > Experiment detail: https://app.optimizely.com/edit?experiment_id=3400641306 > Optimizely experiment: > https://app.optimizely.com/edit?experiment_id=3400641306 > > When setting up the test, I noticed that webpagetest.org reports that the > optimized /new page actually loads more slowly than the original /new page, > at least on the first load. That could give us unexpected results. Any idea > why that might be happening agibson? It's strange, the pull request looks > good to me. Exit out of Optimizely for now and I will look over everything and see how it looks.
I changed the non-Firefox targeting condition from the JS function to the built-in Firefox filter as that works fine now.
In Chrome dev tools on a fresh cache I see: /firefox/new/ - 25 requests, 1.1MB transferred, Finish 2.14s, DOMContentLoaded 957ms, Load 1.78s /firefox/new/2/ - 19 requests, 512KB transferred, Finish 1.5s, DOMContentLoaded 909ms, Load 1.59s
I also adjusted the targeting of the goal to ^https://www.mozilla.org/\w{2,3}(?:-\w{2})?/firefox/new/.*$ because by default it is tied to the URL targeting of the experiment and the experiment is only targeting /firefox/new/ and thus the goal won't be connected to the /firefox/new/2/ page. So I changed the goal targeting to a regex and included .* at the end to capture the /2/. I would have done (\/2/)*, but it should be fine.
Another thing, as for the influence of the optimizely tag on the page, since it is on both the /firefox/new/ and /firefox/new/2/ pages, it will influence both pages equally thus negating the impact to percentage changes between the variations.
(In reply to Alex Gibson [:agibson] from comment #15) > In Chrome dev tools on a fresh cache I see: > > /firefox/new/ - 25 requests, 1.1MB transferred, Finish 2.14s, > DOMContentLoaded 957ms, Load 1.78s > /firefox/new/2/ - 19 requests, 512KB transferred, Finish 1.5s, > DOMContentLoaded 909ms, Load 1.59s That's better!
I also changed the variation code to: /* _optimizely_redirect=https://www.mozilla.org/firefox/new/2/ */ window.location.replace("https://www.mozilla.org/firefox/new/2/"); The other redirect code is for taking the original page's query parameters and passing them along, but since the experiment is targeted at no parameters, this won't happen. :)
:agibson: the only code in the experiment is in comment 19. Do you need a gist for one line of code to review? :jpetto: can you review the experiment that :openjck put together? https://app.optimizely.com/edit?experiment_id=3400641306
Flags: needinfo?(jon)
Flags: needinfo?(agibson)
> I also adjusted the targeting of the goal to > ^https://www.mozilla.org/\w{2,3}(?:-\w{2})?/firefox/new/.*$ because by > default it is tied to the URL targeting of the experiment and the experiment > is only targeting /firefox/new/ and thus the goal won't be connected to the > /firefox/new/2/ page. So I changed the goal targeting to a regex and > included .* at the end to capture the /2/. I would have done (\/2/)*, but it > should be fine. > > ... > > I also changed the variation code to: > > /* _optimizely_redirect=https://www.mozilla.org/firefox/new/2/ */ > window.location.replace("https://www.mozilla.org/firefox/new/2/"); > > The other redirect code is for taking the original page's query parameters > and passing them along, but since the experiment is targeted at no > parameters, this won't happen. :) Great catches. Thanks!
(In reply to Chris More [:cmore] from comment #19) > I also changed the variation code to: > > /* _optimizely_redirect=https://www.mozilla.org/firefox/new/2/ */ > window.location.replace("https://www.mozilla.org/firefox/new/2/"); > > The other redirect code is for taking the original page's query parameters > and passing them along, but since the experiment is targeted at no > parameters, this won't happen. :) I don't think we need a gist for review for this one-liner, thanks Chris
Flags: needinfo?(agibson)
(In reply to Chris More [:cmore] from comment #17) > Another thing, as for the influence of the optimizely tag on the page, since > it is on both the /firefox/new/ and /firefox/new/2/ pages, it will influence > both pages equally thus negating the impact to percentage changes between > the variations. I'm not sure if this sounds quite right? (unless I'm misinterpreting something) A user who ends up on /firefox/new/2/ will go through the following conditions before seeing the Optimized page: 1.) User lands on /firefox/new/ 2.) Browser hits and parses optimizely.js 3.) Browser waits for the network & server response while it gets redirected to /firefox/new/2/ 4.) User lands on /firefox/new/2/ 5.) Browser hits and parses optimizely.js for a *2nd time* 6.) Render-blocking optimizely.js script finishes, and they only then see the optimized page. Having to wait on the network for a client side redirect can already have a negative impact. It seems to me like making the user hit optimizely.js twice before seeing the page will negatively impact things further?
(In reply to Alex Gibson [:agibson] from comment #23) > (In reply to Chris More [:cmore] from comment #17) > > Another thing, as for the influence of the optimizely tag on the page, since > > it is on both the /firefox/new/ and /firefox/new/2/ pages, it will influence > > both pages equally thus negating the impact to percentage changes between > > the variations. > > I'm not sure if this sounds quite right? (unless I'm misinterpreting > something) > > A user who ends up on /firefox/new/2/ will go through the following > conditions before seeing the Optimized page: > > 1.) User lands on /firefox/new/ > 2.) Browser hits and parses optimizely.js > 3.) Browser waits for the network & server response while it gets redirected > to /firefox/new/2/ > 4.) User lands on /firefox/new/2/ > 5.) Browser hits and parses optimizely.js for a *2nd time* > 6.) Render-blocking optimizely.js script finishes, and they only then see > the optimized page. > > Having to wait on the network for a client side redirect can already have a > negative impact. It seems to me like making the user hit optimizely.js twice > before seeing the page will negatively impact things further? It seems possible that it will have some impact to a redirect test where we are testing load time, but I just want to confirm that the conversions are roughly as good as the existing page. We all can imagine that a lighter quicker page is better and that there isn't much advantage to slower pages and thus it could not be worth testing. The main reason why I wanted to test this new page as an A/B test is to ensure we didn't have any regressions and that conversion rates look about as good as the current page on various different segements/browsers. I will be able to segment the results in Optimizely by source/browser and see if there are any differences. If the /new/2/ page looks to be performing similar to the current /new/ page, we'll push forward for a release. I don't need to win major wins in conversion rates of this page to consider it success. Anything that makes a page lighter/quicker and later less complex, sounds good to me. Does that work?
(In reply to Chris More [:cmore] from comment #24) > It seems possible that it will have some impact to a redirect test where we > are testing load time, but I just want to confirm that the conversions are > roughly as good as the existing page. We all can imagine that a lighter > quicker page is better and that there isn't much advantage to slower pages > and thus it could not be worth testing. The main reason why I wanted to test > this new page as an A/B test is to ensure we didn't have any regressions and > that conversion rates look about as good as the current page on various > different segements/browsers. I will be able to segment the results in > Optimizely by source/browser and see if there are any differences. If the > /new/2/ page looks to be performing similar to the current /new/ page, we'll > push forward for a release. I don't need to win major wins in conversion > rates of this page to consider it success. Anything that makes a page > lighter/quicker and later less complex, sounds good to me. > > Does that work? Sounds good to me, thanks Chris
Thanks, :agibson. So, are we all r+ here and good to go to enable?
As soon as we get a r+ from agibson or jpetto, I will enable this test. The areas to review: * URL targeting * Audience targeting * Goal setting * Variation code. Thanks
Only questions from me are regarding the URL regex: ^https://www.mozilla.org/\w{2,3}(?:-\w{2})?/firefox/new/.*$ We should be escaping the literal slashes and dots to be safe, right? ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/.*$ In terms of the URL capturing both /new/ and /new/2/, would this mean that users could be stuck in a redirect loop? If they land on /new/, get redirected to /new/2/, could they again be redirected to /new/2/? I'm probably not fully understanding how the experiment will work. For capturing /new/2/, should we be more precise just in case? Would be easy to add: ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/(2\/)?$
Flags: needinfo?(jon)
Changed URL targeting to: ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/$ (no trailing parameters and won't capture /firefox/new/2/) Goal Targeting: ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/(2\/)?$ (on /firefox/new/ or /firefox/new/2/ with no trailing parameters) Thanks jpetto
(In reply to Jon Petto [:jpetto] from comment #28) > Only questions from me are regarding the URL regex: > > ^https://www.mozilla.org/\w{2,3}(?:-\w{2})?/firefox/new/.*$ > > We should be escaping the literal slashes and dots to be safe, right? > > ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/.*$ > > In terms of the URL capturing both /new/ and /new/2/, would this mean that > users could be stuck in a redirect loop? If they land on /new/, get > redirected to /new/2/, could they again be redirected to /new/2/? I'm > probably not fully understanding how the experiment will work. > What you mentioned above was for goal targeting, not the URL targeting. The URL targeting was always ./new/$ so that it didn't get executed on /firefox/new/2/ and we send people on loops. Goal targeting can be a bit more open and less restrictive, because the only people that can get into the experiment are people who are in the URL targeting and the URL targeting was only ./new/$. > For capturing /new/2/, should we be more precise just in case? Would be easy > to add: > > ^https:\/\/www\.mozilla\.org\/\w{2,3}(?:-\w{2})?\/firefox\/new\/(2\/)?$ As mentioned above, we don't need to be as precise about the (2\/)?, but it doesn't hurt and makes it more explicit. Escaping :openjck's original regex makes sense and what I try to do too. Thanks.
Ah, ok. I figured I was missing something with that goal/URL targeting. Escaped URLs look good to me!
Sounds good! Once we get the regression noted https://bugzilla.mozilla.org/show_bug.cgi?id=1190614#c37 pushed to prod, we should be good to go.
Regression fix has made it to prod: https://www.mozilla.org/en-US/firefox/new/2/
(In reply to Jon Petto [:jpetto] from comment #33) > Regression fix has made it to prod: > > https://www.mozilla.org/en-US/firefox/new/2/ Looks good. Thanks everyone.
Test enabled at 9/8/2015 @ 12:00pm PST. Will be monitoring results.
Test stopped, localized pages are 404ing on bedrock. Comment added here: https://bugzilla.mozilla.org/show_bug.cgi?id=1190614#c38 Going to run the test only for en-US until this is resolved.
Test running for en-US locale only until localized versions of /firefox/new/2/ are available.
Update on the tests. Here's the update on the A/B tests by a few important countries: * United States +3.1% @ 99% confidence * India +1.8% @ 64% confidence * Non-United States Non-India: +2.8% @ 99% confidence It's looking good!
(In reply to Chris More [:cmore] from comment #38) > Update on the tests. > > Here's the update on the A/B tests by a few important countries: > > * United States +3.1% @ 99% confidence > * India +1.8% @ 64% confidence > * Non-United States Non-India: +2.8% @ 99% confidence > > It's looking good! Awesome, thanks Chris!
(In reply to Alex Gibson [:agibson] from comment #39) > (In reply to Chris More [:cmore] from comment #38) > > Update on the tests. > > > > Here's the update on the A/B tests by a few important countries: > > > > * United States +3.1% @ 99% confidence > > * India +1.8% @ 64% confidence > > * Non-United States Non-India: +2.8% @ 99% confidence > > > > It's looking good! > > Awesome, thanks Chris! That is super exciting, Cmore! As you and I have discussed, I'm going to meet with Winston today to better understand requirements for a download experience for the Fall campaign, and then I'd like to move forward with the revisions documented here https://bugzilla.mozilla.org/show_bug.cgi?id=1187057 for Nov 2.
(In reply to Chris More [:cmore] from comment #38) > Update on the tests. > > Here's the update on the A/B tests by a few important countries: > > * United States +3.1% @ 99% confidence > * India +1.8% @ 64% confidence > * Non-United States Non-India: +2.8% @ 99% confidence > > It's looking good! Woo! Thanks for putting the test together :cmore!
Test complete! Here's the summary of the results: United States: +3.4% India: +3.7% Other: +3.4% In the US: IE: +4.1% Chrome: No change Safari: +13.1% (woah!) So, it appears to be a global win and is mainly driven by IE conversion rates. Will take an action to update the mana page with the results.
Flags: needinfo?(chrismore.bugzilla)
Flags: needinfo?(chrismore.bugzilla)
The winning optimized /new page is now in production, so closing this bug.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.