923575 - Make UA override update more scalable and robust

Status: NEW

(Core :: Networking: HTTP, defect, P5)

Description

[(inactive) Jim Chen [:jchen] [:darchons]]

From bug 897221,

(In reply to Andreas Gal :gal from comment #35)
> I would like to see a follow-up bug with a better scaling approach. UA
> issues will be with us for a while. Even if we focus on the top N sites in
> top K locales, we can easily grow a 50k list just to address the most urgent
> UA issues.
> 
> Scaling issues:
> - we don't want to keep this in memory all the time (database in indexeddb
> or even better a mmapped sorted list?)
> - we don't want to re-download the whole list all the time (we should factor
> out the download code into a generic incremental update-capable download
> path for lists like this and others)
> - we don't want to slow down network requests with the lookup (maybe add a
> bloom filter that helps us preselect where to check for overrides at all)
> - some devices don't care about UAs for some locales, so how about a per-TLD
> list that is downloaded the first time you access that TLD?
> 
> We don't need to address all of this at once, but lets have a clear plan.

(In reply to Andreas Gal :gal from comment #36)
> Last bit I forgot: bonus points if we can come up with a way to detect the
> top N sites. This will need some solid privacy reviewing etc but it would
> definitely help maintaining the UA override list. If we know the top N sites
> people go to in a locale, we can do some analysis and see if we get the
> right mobile content with the FFOS vs WebKit UA and then focus evangelism
> and overrides as needed.

Comment 1

[Lawrence Mandel [:lmandel] (use needinfo)]

I think the two issues from the two comments are really two separate issues. I'd prefer to see a separate bug for the top sites suggestion as that is going to require privacy and legal review before we can proceed.

Comment 2

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258