Mapper should yield content when returning full mapfiles

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
4 years ago
a year ago

People

(Reporter: pmoore, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2743] )

(Reporter)

Description

4 years ago
See https://bugzilla.mozilla.org/show_bug.cgi?id=847640#c30 for context.

Basically, mapper aggregates the full mapfile in memory, retrieving content from the database, and only when it has created the full response content in-memory, it starts to stream it back to the requester. Instead, it should write back to the http response as it gets data from the database, so there is not a significant delay between making a request, and starting to receive content.

This is important, since after 30s of inactivity, the load balancer will cut the connection and return an HTTP 500 response.
We should verify, but I think that the load balancers give up after 30 seconds of inactivity, so as long as data is flowing, things are fine.

Flask does make streaming pretty easy, thankfully:

http://flask.pocoo.org/docs/patterns/streaming/
However, the mapfile data is sorted, so streaming may not reduce time-to-first-byte.
(Reporter)

Comment 3

4 years ago
(In reply to Hal Wine [:hwine] (use needinfo) from comment #2)
> However, the mapfile data is sorted, so streaming may not reduce
> time-to-first-byte.

There is a delay from querying the DB until starting to receive rows back via the database connection, and there is a second delay from receiving the first data back from the database connection until the full mapfile is built up in memory. I think we can impact this second delay with streaming, but you are quite right Hal that the first part also incurs a delay. I'm not sure how much is the first part, and how much is the second part, at the moment.

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2736]

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2736] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2743]
(Reporter)

Updated

3 years ago
Assignee: nobody → hwine
no one to work on this at the moment

If anyone is seeing this in the real world, could they add details to help with prioritization.
Assignee: hwine → nobody
(Reporter)

Comment 5

3 years ago
No need to touch this. No problems in over a year of use, and not a system we need to heavily invest in, especially with deprioritisation of B2G which was main user.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

a year ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.