Closed Bug 1187057 Opened 9 years ago Closed 7 years ago

/firefox/new/ near-term improvements and future strategy

Categories

(www.mozilla.org :: Pages & Content, defect)

Production
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jpetto, Unassigned)

References

()

Details

(Whiteboard: [kb=1806973][fxgrowth])

The /firefox/new/ page is at a point of complexity where changes are prohibitively difficult and end user experience is being negatively affected. A performance upgrade and new strategy are needed.

# Current issues:

- As the page must work in all browsers back to IE 6, code changes are risky and require heavy QA. Change requests come in frequently enough that this is a problem.
- Difficult to offer conditional messaging and run A/B tests, as requires adding logic/complexity/weight to the template. This negatively impacts users, developers, and org growth.
- Current UI/UX offers very little room for alternate messaging (limited height), and adds considerable complexity for marginal value (scene 2).

# Possible near-term improvements:

## Simplify & streamline

- Remove scene 2, replace install instructions with SUMO link
	- Install instructions are out of date, do not support Linux, and are difficult to maintain (Windows 10 is right around the corner)
- Move JS to footer (non-blocking)
	- Was in header for GA, but looks unnecessary now due to GTM switch?
- Clean up/audit template conditionals (e.g. l10n_has_tag)

## Implement new strategy for alternate views

- There should be a single, canonical /firefox/new/ page. This page will be simple, lightweight, and bulletproof back to IE 6. This page will be changed very infrequently.
- Variations (A/B tests, campaign/referrer specific messaging, etc) should be done through URL parameters, which will trigger a different template to be sent to the user. This will keep logic out of the templates, resulting in faster load times, better performance, easier maintenance, and higher iteration/dev speed.

### Open/future questions/concerns:

- Do we still need to handle scene 2 related URL params due to existing inbound links?
- How would A/B tests run without making an extra HTTP request yet still leverage cache?
- Okay with possibility of many similar (non-DRY) templates? 
- Will SEO be negatively affected if completely different templates can be served to a single URL? What about using new URLs (e.g. /firefox/download/*) for test/campaign download pages?
- Should we still cater for non-up-to-date messaging? We try and display messaging depending on if a user is up to date or not, but this has a long history of being unreliable and inaccurate (e.g. ESR releases being difficult to detect). We seem to go to great efforts to prevent people from downloading Firefox in these situations - is it worth it? is it actually a UX anti-pattern?
- How can we better handle older browsers where the download can't be triggered automatically? Should we keep on bending backwards to support them, or just use more prominent messaging? e.g. "If your download does not start automatically, click here"? (the current text is tiny).
- Could we explore other (better) methods to trigger the download automatically? (i.e. not relying on a click). We tried an iframe in the past, but non-secure redirects in bouncer prevented this.

(Thanks to :agibson & :jgmize for help putting together the above.)
Whiteboard: [kb=1806973]
Whiteboard: [kb=1806973] → [kb=1806973][fxgrowth]
See Also: → 1182448
A tweet for reference re: "latest version" messaging on /firefox/new being an anti-pattern:

https://twitter.com/rmill/status/626791995852333057

Fwiw, personally I agree what we are doing currently feels odd. Do we try too hard to stop people downloading Firefox, by assuming we're acting in their best interest?
It is a tough balance. We try hard because it is bad for them if they don't need it, as well as expensive for us. I think that tweet is from a person in the incredible minority, but I've no data to back that up. 

That said I do think we should make the "download anyway" button much more obvious and prominent.
(In reply to Paul McLanahan [:pmac] from comment #2)
> That said I do think we should make the "download anyway" button much more
> obvious and prominent.

Agreed. I'm not suggesting we show everyone the default messaging/download button, but it caused this person enough confusion to tweet about it. This is a tech-savy user as well. If someone really wants to download Firefox, we should make sure they still can without too much head-scratching.
(In reply to Alex Gibson [:agibson] from comment #3)
> If someone really wants to download Firefox, we
> should make sure they still can without too much head-scratching.

THIS! Very well said.
Let's write user stories here in this doc on what the page should be doing:

https://docs.google.com/document/d/1NQqrhUdRkPWfOXIGUpcMVACG2Kqzzaa1R1JUP4G-n7k/edit?usp=sharing
Hi jpetto, agibson, jgmize, pmac, schalk, others-

Cmore and I have reviewed the user stories and proposed some solutions/asked some questions/priortized here:https://public.etherpad-mozilla.org/p/firefox-new-user-stories-2015  (sorry about the formatting, gdocs is down today)

Please review and ask any questions/propose alternate solutions - especially lines 45-52 - Monday, October 19.  Let me know if you want me to schedule time for us to discuss.

Thanks,
Jen
Flags: needinfo?(schalk.neethling.bugs)
Flags: needinfo?(pmac)
Flags: needinfo?(jon)
Flags: needinfo?(agibson)
Flags: needinfo?(jmize)
(In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> Hi jpetto, agibson, jgmize, pmac, schalk, others-
> 
> Cmore and I have reviewed the user stories and proposed some solutions/asked
> some questions/priortized
> here:https://public.etherpad-mozilla.org/p/firefox-new-user-stories-2015 
> (sorry about the formatting, gdocs is down today)
> 
> Please review and ask any questions/propose alternate solutions - especially
> lines 45-52 - Monday, October 19.  Let me know if you want me to schedule
> time for us to discuss.
> 
> Thanks,
> Jen

I left some comments in purple, thanks
Flags: needinfo?(agibson)
Added some comments in orange.
Flags: needinfo?(jon)
Thanks Jon and Alex.  Cmore and I will take a look on Friday when we meet.
(In reply to Jennifer Bertsch [:jbertsch] from comment #9)
> Thanks Jon and Alex.  Cmore and I will take a look on Friday when we meet.

Cmore and I made some comments in the etherpad.  Let's discuss whether or not anything else is still blocking this work and who has bandwidth to start it :)
(In reply to Jennifer Bertsch [:jbertsch] from comment #10)
> (In reply to Jennifer Bertsch [:jbertsch] from comment #9)
> > Thanks Jon and Alex.  Cmore and I will take a look on Friday when we meet.
> 
> Cmore and I made some comments in the etherpad.  Let's discuss whether or
> not anything else is still blocking this work and who has bandwidth to start
> it :)

I have read through the etherpad but I guess I am a bit to out of the loop to gather all the action items. I guess the one's with a decision is are the action items?

Is it possible to update the bug with those? Thanks!
Flags: needinfo?(schalk.neethling.bugs)
I think this is done. Clearing my NI.
Flags: needinfo?(pmac)
Hi Pmac and Alex-

I think we have one remaining open question here for the two of you to address before we can begin work on this one:

"Do we use Optimizely or our own js to manage the proposed "different templates for different purposes and a/b testing playgrounds that can't break /new/".  I'm happy to not use Optimizely if the team is willing to manage the js."

See lines 51-61 here for additional background information.

Thx,
Jen
Flags: needinfo?(pmac)
Flags: needinfo?(jmize)
Flags: needinfo?(agibson)
(In reply to Jennifer Bertsch [:jbertsch] from comment #13)
> Hi Pmac and Alex-
> 
> I think we have one remaining open question here for the two of you to
> address before we can begin work on this one:
> 
> "Do we use Optimizely or our own js to manage the proposed "different
> templates for different purposes and a/b testing playgrounds that can't
> break /new/".  I'm happy to not use Optimizely if the team is willing to
> manage the js."
> 
> See lines 51-61 here for additional background information.
> 
> Thx,
> Jen

This is really an open solution that depends on both available time and resources to carry out the work necessary. We've only really touched on what the requirements would be in this discussion, let alone proposed a solution for how we would implement it, so it's not something I feel I can provide a resolution for in the space of a comment. It would be great to mitigate using Optimizely altogether, but it's not something that can really be lumped together as a small part of an existing project. It really warrants being a mini-project of its own, and is not something I think can or should block the rest of the work being carried out.
Flags: needinfo?(agibson)
Alex is exactly right. We want to remove as much dependence on 3rd party JS as possible, but it will be a lot of work. For now Optimizely is what we have unfortunately, so we just need to stick with the process that we've developed and only enable the Optimizely code snippet on the pages we need, and then only for exactly the time we need to mitigate as much of the security risk as we can.
Flags: needinfo?(pmac)
+1 with sticking with the process that we have as it is the only realistic solution given the large investment in building something from the ground up. If we were to build something, I would want it to do something Optimizely can't do and that's probably integration with the product metrics directly on top of the benefits less 3rd party JS.
(In reply to Chris More [:cmore] from comment #16)
> +1 with sticking with the process that we have as it is the only realistic
> solution given the large investment in building something from the ground
> up. If we were to build something, I would want it to do something
> Optimizely can't do and that's probably integration with the product metrics
> directly on top of the benefits less 3rd party JS.

I agree that optimizely is the only realistic near-term option, but I think a long-term solution with features like direct integration with product metrics may justify the large investment required. I think that discussion is decidedly out of scope for this bug, but :cmore I'd greatly appreciate it if you would spend a little time thinking about the features and metrics that we can't get from optimizely, and perhaps we can have a higher-bandwidth discussion about it later?
Yeah, totally! I could probably list out the features that are used 80% of the time so we can get down to scope that we could spec out the investment. Do you want me to have the growth team list out the most common things we use Optimizely for before having the conversation?
(In reply to Chris More [:cmore] from comment #18)
> Yeah, totally! I could probably list out the features that are used 80% of
> the time so we can get down to scope that we could spec out the investment.
> Do you want me to have the growth team list out the most common things we
> use Optimizely for before having the conversation?

Yes please :)
This is a good article on this very topic with some good insights:

https://www.redfin.com/devblog/2014/02/a-b-testing-build-or-buy.html
Regarding template variations, I think a good place to start would be locale based templates as :pmac outlined in bug 1222137. This would be a great way to have a canonical, stable template (new.html) while giving us an easy way to fork a version (e.g. new.en-US.html) that we can hack for A/B tests, which, based on L10n cost, are generally restricted to a specific locale.

When we get this infrastructure working for /firefox/new/, applying it to other pages/templates will be as simple as naming new templates according to our convention.
(In reply to Jon Petto [:jpetto] from comment #21)
> Regarding template variations, I think a good place to start would be locale
> based templates as :pmac outlined in bug 1222137. This would be a great way
> to have a canonical, stable template (new.html) while giving us an easy way
> to fork a version (e.g. new.en-US.html) that we can hack for A/B tests,
> which, based on L10n cost, are generally restricted to a specific locale.

I think a similar system built specifically for a/b testing could work, but the locale-based one I outlined in the bug wouldn't. If there's a new.en-US.html then it'll always be used for the en-US page and only non-en-US requests would see new.html. I don't think this is what we necessarily want for a/b testing. I think we would usually (always?) want variations within a locale. I don't think any data gathered would be very meaningful if we were comparing users of different translations of the page.

However, we could setup a similarly simple a/b template naming convention for doing things like this if we need it. I know we'd like to do something more robust soon, but something like the following might work in the interim: We could e.g. name a template new.variation-b.html, and that version could be accessible at /new/?v=b.

This is pretty simple and not that powerful, but if you want something easier than custom view creation for now, then this might work.
I think we can close this bug as FIXED.

We now have a lightweight alternative to Optimizely that has proved itself to be robust in production many times [1]. Traffic Cop experiments can also land as part of the same code change in the bedrock repo, which removes much of the the disconnect that happens when reviewing Optimizely code.

We also have the ability to handle locale specific templates, and A/B switching much easier. Much of the technical debt for /firefox/new/ has also been paid off, now that the templates are split.

[1] http://bedrock.readthedocs.io/en/latest/javascript-libs.html#mozilla-traffic-cop
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Is there any chance to look at this? I think it's a relatively easy fix for firefox/new, and it would make our localizers a lot happier.
You need to log in before you can comment on or make changes to this bug.