Closed Bug 1659867 Opened 3 months ago Closed 3 months ago

Messaging system targeting FxA oauth services/devices causes many requests to the FxA attached_devices endpoint


(Firefox :: Messaging System, defect, P1)




82 Branch
82.2 - Sep 7 - Sep 20
Tracking Status
firefox82 --- fixed


(Reporter: jbuck, Assigned: andreio)


(Blocks 1 open bug)



(1 file)

On August 17th, when deploying Firefox Accounts Train 184 to production I noticed elevated CPU/memory usage on the oauth database and redis instances correlating with a 10% increase in HTTP traffic starting at 2020-08-17T00:00:00Z. After digging into the logs, we found that the /v1/account/attached_devices route was seeing up to 100x to 150x more traffic than normal . :rfkelly investigated various experiment platforms that we use and found that ASRouter has the ability to list oauth service services/devices connected to the signed-in Firefox Account to support targeting messages. :k88hudson showed me how to use the ASRouter dev tools which allowed me to find two snippets (24160 and 24163) that use this feature.

:andreio added some more context around this filter:

andreio - I know it was always problematic, we need the data to be as recent as possible (users actives some services => stop showing messages about that services) but the call is expensive to make. Would increasing the cache time on our side help?
jbuck - on the caching time side - how often does snippets check targeting?
andreio - every time you open a new tab until the user interacts with the snippet or the campaign is taken down
andreio - there's a small 5 min cache (between opening 2 tabs) where we don't look at targeting (we assume nothing changed) and a 2hr cache for that attribute
jbuck - lets say it got bumped up to a 4 hour cache - would it be reasonable to assume that it’d halve the traffic to this route? (edited)

So, I think the short-term fix is increasing the cache expiry time to 4 hours, to reduce traffic to this endpoint.

The long-term fix is probably having FxA client within the browser cache this information/use push notifications to keep it up to date so ASRouter doesn't need to worry about these details.

See Also: → 1600779
Assignee: nobody → andrei.br92
Blocks: cfr
Severity: -- → S3
Iteration: --- → 82.1 - Aug 24 - Sep 6
Priority: -- → P1
Iteration: 82.1 - Aug 24 - Sep 6 → 82.2 - Sep 7 - Sep 20
Pushed by
Increase update interval for FxA endpoint to four hours r=jbuck
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 82 Branch
You need to log in before you can comment on or make changes to this bug.