Closed Bug 1176429 (remote-hosted-newtab) Opened 9 years ago Closed 3 years ago

[Meta] Make about:newtab remote

Categories

(Firefox :: New Tab Page, defect)

defect
Not set
normal

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: emtwo, Unassigned)

References

(Depends on 3 open bugs)

Details

(Keywords: meta, Whiteboard: [story][fxgrowth])

We want to make about:newtab remote so that updates to it are not tied to the Firefox release cycle. This means moving the js/css/html under browser/base/content/newtab into the cloud and pre-fetching it before a user opens about:newtab.

When the page is loaded, the user data is injected into it so that pinned, blocked, suggested tiles, etc are displayed as expected.
Note: We also want this page to easily translate on to other browsers. This means we should avoid relying on Firefox-specific data to populate the new tab.
Depends on: 1180251
Summary: Make about:newtab remote → [Meta] Make about:newtab remote
Depends on: 1021654, 1167601
Depends on: 1180283
Depends on: 1180291
Depends on: 1180298
Whiteboard: [story]
Depends on: 1183206
Depends on: 1186121
(In reply to Marina Samuel [:emtwo] from comment #1)
> Note: We also want this page to easily translate on to other browsers. This
> means we should avoid relying on Firefox-specific data to populate the new
> tab.

We want other browsers to be able to use the newtab page? I'm curious, can you link me to any discussions that provide more context for why we want this?
you're looking at the discussion.  I asked for it.  We want to use the web.
What's the user story for non-Firefox browsers?
This could potentially be *very* interesting and I'm excited to what would be possible if it was a web experience.

A few questions.

What about about:home? Or are we thinking that merging about:home and about:newtab into a single experience and put it on the web is the end goal?

Have we explored the ability to A/B about:newtab as it exists in the product now without it having to be on the web? It is possible to do now by creating add-ons that manipulate about:newtab, create multiple add-ons for multiple variations of the A/B test, and package them up as funnelcakes, and distribute them on www.mozilla.org. We are almost doing that now with some recent tests. Though, doing A/B testings and improvements to a web experience is *much* easier than an in-product experiences with the tools that we have now.

So, I would love for about:newtab to be on the web, but it won't be without challenges due to the sheer volume of potential web requests from active users. See also the crazy amount of impressions about:home gets and only a small minority of active users have it set as their homepage.
> What's the user story for non-Firefox browsers?

I'm not sure it's *the* user story for non-Firefox browsers, but I think one of the possibilities that remote newtab page opens up is allowing us to consider things like a Firefox experience in a Chrome add-on.

> What about about:home? Or are we thinking that merging
> about:home and about:newtab into a single experience
> and put it on the web is the end goal?

I heard that UX is doing a telemetry experiment doing just that.
They are sampling a portion of the pre-release population and measuring engagement rates.

IMO, remote new tab page opens up more possibilities for A/B testing / multi-arm bandit experiments. In fact, it becomes so much easier to do more things like incremental roll-outs, and the latency becomes much lower should we want to react or create new changes.

> So, I would love for about:newtab to be on the web, but
> it won't be without challenges due to the sheer volume
> of potential web requests from active users.

We have experience on the server-side building services that can handle the load. Our Tile infrastructure handles 10k req/sec and we've got room to grow, thanks to our engineering and operations teams.

In fact, the remote new tab pages themselves will, in most likelihood, be served from a CDN.

> It is possible to do now by creating add-ons that manipulate about:newtab,
> create multiple add-ons for multiple variations of the A/B test, and package
> them up as funnelcakes, and distribute them on www.mozilla.org.

I'm not familiar with funnelcake (aside from the edible variety) care to elaborate?
+1 on everything from comment 6. It makes me and the Firefox Growth Team [1] very excited. We have hypotheses rolling around right now about that future state. :)

[1] https://wiki.mozilla.org/Growth_Team
As :cmore mentioned, we've spent a good bit of time looking into about:home/ about:newtab related things. We are currently testing a distribution of Firefox with the newtab page set as the default homepage to see how it affects search volume and retention over time (bug 1184279).

Additionally, we've got some really interesting snippet/impression/search projections using Testpilot and FHR data on the homepage. If you have any further questions about data we have gathered with respect to about:home and about:newtab, feel free to ping cmore or myself (fpolizzi) on IRC.
So, I’m pretty excited by the opportunities this offers for optimization, but…

1) I feel like we could get _almost_ all the benefits with an add-on that contained the html, instead of loading it from a server every time, and
2) I’m worried about what will show on the newtab page when I’m at a Starbucks, or behind another captive portal.  Along the same lines, what will happen in offline mode?  Furthermore,
3) I believe all of the “about:” pages are currently built in to Firefox, so changing this one to load from the network is a pretty big violation of our user’s expectations, and doesn’t seem in-line with our usual stance on privacy.

Finally, while I _love_ A/B testing our product, the newtab page (and home page) are very political, seemingly being co-owned by Product, Engagement, and Content Services (and Growth too maybe?), so even if we could run A/B tests on the content, I believe we would need to get a lot of teams in the room to discuss what acceptable results from the tests would be before we ran them, and that seems like a far more limiting factor than whether the page is in content or not.  (Particularly if the content was in an add-on that we could update every day!)

(I'm not subscribing to this bug, because I trust you all can come to a reasonable decision.  If you have any further questions I can help with, feel free to needinfo me.  ;)
So far the discussions seem like they are in the right direction and have touched most of my thoughts.

I would like to add that if the main goal is to optimize the page, we need to be sure we'll be able to track/measure what we need to define success. For example, if we change the content of the page and only measure the change in engagement with that page, we may be neglecting potential impact on overall user churn. How do we tie in a change to a webpage to our churn data? It seems that this is the most important metric of them all. 

We want to improve experience to keep our users for a longer period of time so let's make sure we can measure it or the A/B testing will be useless.
(In reply to Blake Winton (:bwinton) from comment #9)
> 2) I’m worried about what will show on the newtab page when I’m at a
> Starbucks, or behind another captive portal.  Along the same lines, what
> will happen in offline mode?
I believe the layout will be prefetched/cached over key pinned https, so a failed update from captive portal would probably just show the previous layout.
I would very much like to make it work without addons and kill all platform bugs that prevent us from doing this.  The problems that we'll hit are the problems that every web developer will hit.  Offline *has* to work.
(In reply to Blake Winton (:bwinton) from comment #9)
> 1) I feel like we could get _almost_ all the benefits with an add-on that
> contained the html, instead of loading it from a server every time

This is indeed an option we considered. And we do still want to have part of newtab be inside an addon - the part that handles user data. This cannot be remote for privacy reasons, but we still want the ability to make frequent updates.

Making the html/css/js remote can allow quick/easy changes of the look and feel of newtab even faster than the addon update cycle would allow.

> 2) I’m worried about what will show on the newtab page when I’m at a
> Starbucks, or behind another captive portal. Along the same lines, what
> will happen in offline mode?

This is a very interesting and valid point that we need to think about. Currently, we're looking into using AppCache to handling offline mode (bug 1180291) but if we design a custom caching method we can always use a cached version even if we're behind a captive portal.


> 3) I believe all of the “about:” pages are currently built in to Firefox, so
> changing this one to load from the network is a pretty big violation of our
> user’s expectations, and doesn’t seem in-line with our usual stance on
> privacy.

Actually, the only parts of newtab that will be remote will be the HTML/JS/CSS that runs it. The user data backing it remains in Firefox and the page is only injected with the data after it's been downloaded. This is much like how FHR works.

> Finally, while I _love_ A/B testing our product, the newtab page (and home
> page) are very political, seemingly being co-owned by Product, Engagement,
> and Content Services (and Growth too maybe?), so even if we could run A/B
> tests on the content, I believe we would need to get a lot of teams in the
> room to discuss what acceptable results from the tests would be before we
> ran them, and that seems like a far more limiting factor than whether the
> page is in content or not.  (Particularly if the content was in an add-on
> that we could update every day!)

A/B testing is only one of the benefits of making newtab remote. Not every change we make will need A/B testing (e.g. We want to make the corners of tiles 2px more rounded)
(In reply to Marina Samuel [:emtwo] from comment #13)
> Making the html/css/js remote can allow quick/easy changes of the look and
> feel of newtab even faster than the addon update cycle would allow.

Assuming users perceive the newtab as part of the built-in UI, too many rapid changes might be disconcerting to them though.

> > 2) I’m worried about what will show on the newtab page when I’m at a
> > Starbucks, or behind another captive portal. Along the same lines, what
> > will happen in offline mode?
> 
> This is a very interesting and valid point that we need to think about.
> Currently, we're looking into using AppCache to handling offline mode (bug
> 1180291) but if we design a custom caching method we can always use a cached
> version even if we're behind a captive portal.

Fetching the page only at the point where a user actually opens a newtab would probably not feel snappy, so you'd probably be showing a cached version most of the time anyway?
(In reply to aleth [:aleth] from comment #14)
> (In reply to Marina Samuel [:emtwo] from comment #13)
> > Making the html/css/js remote can allow quick/easy changes of the look and
> > feel of newtab even faster than the addon update cycle would allow.
> 
> Assuming users perceive the newtab as part of the built-in UI, too many
> rapid changes might be disconcerting to them though.
> 
> > > 2) I’m worried about what will show on the newtab page when I’m at a
> > > Starbucks, or behind another captive portal. Along the same lines, what
> > > will happen in offline mode?
> > 
> > This is a very interesting and valid point that we need to think about.
> > Currently, we're looking into using AppCache to handling offline mode (bug
> > 1180291) but if we design a custom caching method we can always use a cached
> > version even if we're behind a captive portal.
> 
> Fetching the page only at the point where a user actually opens a newtab
> would probably not feel snappy, so you'd probably be showing a cached
> version most of the time anyway?

Mobile apps usually pull the A/B test content at launch and then use the cached content until the app is restarted and checks to see if a newer version is available or if the test is done. This allows to have the best performance possible. Perhaps something similar could be done. (best of the web and app A/B testing worlds)
Summary: [Meta] Make about:newtab remote → (remote-hosted-newtab) [Meta] Make about:newtab remote
Alias: remote-hosted-newtab
Summary: (remote-hosted-newtab) [Meta] Make about:newtab remote → [Meta] Make about:newtab remote
Depends on: 1193856
Depends on: 1193858
Depends on: 1193862
Depends on: 1193865
Depends on: 1193867
Depends on: 1194236
Depends on: 1195494
Depends on: 1195495
Depends on: 1196755
Depends on: 1196758
Depends on: 1196762
Depends on: 1196770
Depends on: 1196773
No longer depends on: 1196758
Whiteboard: [story] → [story][fxgrowth]
Depends on: 1199640
Depends on: 1200920
Depends on: 1201147
Depends on: 1201977
Depends on: 1066787
Depends on: 1206799
Depends on: 1209627
Depends on: 1210053
Depends on: 1204983
Depends on: 1210410
No longer depends on: 1066787
Depends on: 1210478
Depends on: 1210936
Depends on: 1210940
Depends on: 1210944
Depends on: 1212087
Depends on: 1212897
Depends on: 1213078
Depends on: 1214287
Depends on: 1214376
Depends on: 1215357
Depends on: 1215613
Depends on: 1218992
Depends on: 1218996
Depends on: 1220233
Depends on: 1224950
No longer depends on: Content-Signature
(Sorry for jumping in)

Have we discussed the security model against the secured remote newtab content? The "local newTab" content has privilege permission so it can access internal APIs. For remote newTab, what I learned from Bug 1239116, there will be a new Web API to expose gecko implementation. If it's true, is the new API public for all content or only available for signed content?

Thanks :)
Sorry for jumping in again.

Reading Bug 1218998, it looks like the original plan was creating and exposing some Web APIs to specific domains but then changed to use WebChannel.jsm (content/chrome communication only on specific domain).

I wonder if this is the long term plan or just a workaround and be waiting for some new security model.
Hi,

Could I have your opinion regarding comment 16 and comment 17? 

In theory, we are able to securely expose certain powerful APIs (e.g. history/pinned pages/...) to hosted newtab content since it would be protected by content signature and enforced SRI. There's no need to limit those APIs visibility to specific domain. Instead, they only needs to be limited to signed content. 

Thanks!
Flags: needinfo?(ptheriault)
Flags: needinfo?(oyiptong)
Flags: needinfo?(franziskuskiefer)
Hi Henry,
in general we could allow signed content to access some privileged APIs. But that would only work if content-signing is a pure Firefox internal thing. The idea behind the content-signature header though is to allow servers to add additional protection to their page when a CDN holds the TLS key (such that the website provider has no way to detect/defend against malicious CDNs). That's interesting for the general internet population (at least the paranoid one;) such that allowing signed content access to privileged things is probably not a good idea.
Flags: needinfo?(franziskuskiefer)
(In reply to Henry Chang [:henry] from comment #16)

I see your point :henry.

However, content signature is intended to be an additional validation, rather than the only validation to grant elevated privileges.

Granted, the signature does ensure the integrity of the content received. But it's not enough. We need additional security measures, e.g. limiting the privileges granted to a specific origin and/or url scope.

I'd like to echo :franziskus's sentiment.

Here's a counter-example to using content-signature as the sole authorization mechanism:

If an attacker breaks content-signature, they would get access to the keys to the kingdom. We can diminish the exposure by requiring other checks.

There are other aspects to content security that we'd need to ensure. You hinted on that topic by mentioning SRI.

Even if we do enforce SRI, there are still risks involved: there may be vulnerabilities introduced in a sub-resource that we are unaware of. Depending on the situation, we'd need to enforce CSP as well, for instance, to mitigate XSS, but that's hard to enforce.

We also need to ensure HTTPS and key-pinning. If someone compromised our signature key unbeknownst to us, they could in theory inject compromised, but signed data into Firefox.

A vulnerability is bound to happen. A means to restrict the the amount of damage is warranted.
Flags: needinfo?(oyiptong)
(In reply to Henry Chang [:henry] from comment #17)
> I wonder if this is the long term plan or just a workaround and be waiting
> for some new security model.

We were originally using RemotePageManager for chrome/content communications, but we've found that there's a significant vulnerability: RPM allows content to export functions to chrome. While there is X-Ray vision, there is sufficient risk that we decided not to use it.

We then tried to use Web APIs to communicate with remote content. While this would've worked, there were a couple of flaws, mainly that it prevents us from shipping our code as system addons in the future.

* the requirement for native code (via IDL declaration) makes it hard to ship system addons. While it may be possible, it isn't done yet, plus the addon wouldn't be restartless
* shipping web API declarations themselves in system addons might not be possible

WebChannel.jsm, like RemotePageManager, is an abstraction over the message manager infrastructure. Importantly, it is more limited and doesn't have the flaw RPM has.

The plan is to use WebChannel's origin-based communication for now. We'll re-evaluate later if we want to get more granular, adding url scopes on top of the origin-based restriction.
Hi Franziskus, Olivier:

Thanks for your quick reply :) 

> Even if we do enforce SRI, there are still risks involved: there may be vulnerabilities introduced in a 
> sub-resource that we are unaware of. Depending on the situation, we'd need to enforce CSP as well, for 
> instance, to mitigate XSS, but that's hard to enforce.

Since we enforce SRI on every sub-resource, how could it be possible that a non-reviewed sub-resource is introduced? While signing the content, we are able to review all the sub-resource as well.

> We also need to ensure HTTPS and key-pinning. If someone compromised our signature key unbeknownst to us,
> they could in theory inject compromised, but signed data into Firefox.

I was thinking the assumption is the signing server would never be compromised. If it happened, content-signature and SRI is indeed totally useless :(
(In reply to Henry Chang [:henry] from comment #22)
> Since we enforce SRI on every sub-resource, how could it be possible that a
> non-reviewed sub-resource is introduced? While signing the content, we are
> able to review all the sub-resource as well.

Unfortunately it is possible. Some unintentional vulnerabilities may be introduced while still passing review. We are not infallible. Another scenario is when the code isn't ours but is somewhat "trusted" to do the right thing: e.g. your pick of JS libraries: jquery, react, angular, d3, etc.
(In reply to Olivier Yiptong [:oyiptong] from comment #23)
> (In reply to Henry Chang [:henry] from comment #22)
> > Since we enforce SRI on every sub-resource, how could it be possible that a
> > non-reviewed sub-resource is introduced? While signing the content, we are
> > able to review all the sub-resource as well.
> 
> Unfortunately it is possible. Some unintentional vulnerabilities may be
> introduced while still passing review. We are not infallible. 

But how do we deal with add-on review? The add-on has more privilege so I suppose there's more secured model to keep the add-on as safe as possible.

> Another scenario is when the code isn't ours but is somewhat "trusted" to do the
> right thing: e.g. your pick of JS libraries: jquery, react, angular, d3, etc.

Regarding the external scripts issue, we can refuse to sign the content which uses external scripts unless we certainly know they are safe.

Furthermore, we can put a list of granted resources in the html or HTTP header. Only those specified resources could be loaded to ensure the scope of security.
(In reply to Franziskus Kiefer [:fkiefer or :franziskus] from comment #19)
> Hi Henry,
> in general we could allow signed content to access some privileged APIs. But
> that would only work if content-signing is a pure Firefox internal thing.
> The idea behind the content-signature header though is to allow servers to
> add additional protection to their page when a CDN holds the TLS key (such
> that the website provider has no way to detect/defend against malicious
> CDNs). That's interesting for the general internet population (at least the
> paranoid one;) such that allowing signed content access to privileged things
> is probably not a good idea.

Hi Franziskus,

I just studied "HTTPS in CDN" for a little bit [1] so I'd guess what you mentioned is "sharing the private key" to solve the certificate mismatch issue (would occur when using the DNS request routing CDN solution). In this case, if the CDN server is compromised, could we just revoke the certificate associated with the stolen key? If the private key leaks, the content could be modified to use "WebChannel", which is going to be chosen to communicate with internal APIs, to do any evil thing. It doesn't seem to be safer.

[1] http://netsec.ccert.edu.cn/duanhx/files/2012/05/https-in-cdn.pdf
(In reply to Henry Chang [:henry] from comment #24)
> But how do we deal with add-on review? The add-on has more privilege so I
> suppose there's more secured model to keep the add-on as safe as possible.

You are correct, add-ons have access to much more. While we do review the addons, the users, we hope,  expressly chose to use  them. If they didn't, like with spam ware, we actively try to fight the nefarious add-ons.

We are actively building web extensions, which I heard makes add-ons less capable. I haven't made any add-ons for Chrome, but I think the capabilities are much more restricted.

> Regarding the external scripts issue, we can refuse to sign the content
> which uses external scripts unless we certainly know they are safe.

I'm uncomfortable to say that something is safe. I can say that something I wrote looks reasonably safe, after a review, but we never know where a vulnerability may come from. Best be paranoid in that situation and have mitigation strategies.

Perhaps it is just me, but I can't say I can be certain about external code being "safe", unless the code is extremely small. I'd rather assume that external code is "reasonably" safe given certain precautionary measures.

Having options is good to mitigate potential attack vectors because we don't know where they may come from.

However, we do want to build a product at the end of the day, so there's a certain pragmatism also required. Is there something you want to achieve in particular? The specifics might be helpful.

> Furthermore, we can put a list of granted resources in the html or HTTP
> header. Only those specified resources could be loaded to ensure the scope
> of security.

The list of vetted resources is given by SRI, or am I missing something else?
The HTTP header won't work, because the content signature is only concerned about the body.
Hi Oliver,

I really appreciate your feedback :)

Here is the story: For the past half year, we have been working on a new security model [1] proposed by Jonas. The new model is to pack the contents/resources, sign the packed package and publish it. Every resource would have an URL relative to the package URL. (e.g. https://foo.com/app.pak!//index.html) The signed and packaged content could have more privilege (by declaring them in the manifest). 

We've got a working prototype on B2G and had a presentation in mozlando. However, after gathering some feedback from gaia developers, we are trying to "unpack" the package since it seems to be the most unacceptable part from the "webby" perspective.

Lately we learned remote-newtab, content signature and so on. We found that the unpacked "new security model" (if we can figure out one) meets remote-newtab requirements very much. I wrote up a proposal [2] based on the concept of content-signature and SRI. Some of the design might be able to align to the standard. (once they are ready)

> The list of vetted resources is given by SRI, or am I missing something else?

There's no explcit list of sub-resource in SRI specification [3] so we don't have a scope to review. (the script tags in HTML is not enough since script could load other script)

[1] https://wiki.mozilla.org/FirefoxOS/New_security_model
[2] https://docs.google.com/presentation/d/171xW1vbkwk1q_LHbzYUUauUIqeLdt07ScfnUA1ep2BI/edit#slide=id.p
[3] However, there's something similar in CSP level 2: https://www.w3.org/TR/CSP2/#source-list-syntax
(In reply to Henry Chang [:henry] from comment #27)
> Here is the story: For the past half year, we have been working on a new
> security model [1] proposed by Jonas. The new model is to pack the
> contents/resources, sign the packed package and publish it. Every resource
> would have an URL relative to the package URL. (e.g.
> https://foo.com/app.pak!//index.html) The signed and packaged content could
> have more privilege (by declaring them in the manifest).

Sounds exciting!

> We've got a working prototype on B2G and had a presentation in mozlando.
> However, after gathering some feedback from gaia developers, we are trying
> to "unpack" the package since it seems to be the most unacceptable part from
> the "webby" perspective.

Ah yeah, the bandwidth requirements could be wasteful, depending on the use-case.
Also cache-ability could be an issue, if one doesn't need the whole package.

> There's no explcit list of sub-resource in SRI specification [3] so we don't
> have a scope to review. (the script tags in HTML is not enough since script
> could load other script)

You are correct. The plan for remote-newtab is to create this list at the pages' build time, before deployment. There is no way to enforce SRI as of yet.

> Lately we learned remote-newtab, content signature and so on. We found that
> the unpacked "new security model" (if we can figure out one) meets
> remote-newtab requirements very much. I wrote up a proposal [2] based on the
> concept of content-signature and SRI. Some of the design might be able to
> align to the standard. (once they are ready)

Excellent! I'll take a read. We should chat more about this :-)
We currently have a signing/deployment flow. Perhaps it would be a good idea to sync up.
Depends on: 1258728
Discussed elsewhere so removing flag.
Flags: needinfo?(ptheriault)
Keywords: meta
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.