Closed Bug 1020554 Opened 7 years ago Closed 5 years ago

Mapper should yield content when returning full mapfiles


(Release Engineering :: General, defect)

Not set


(Not tracked)



(Reporter: pmoore, Unassigned)


(Whiteboard: [kanban:engops:] )

See for context.

Basically, mapper aggregates the full mapfile in memory, retrieving content from the database, and only when it has created the full response content in-memory, it starts to stream it back to the requester. Instead, it should write back to the http response as it gets data from the database, so there is not a significant delay between making a request, and starting to receive content.

This is important, since after 30s of inactivity, the load balancer will cut the connection and return an HTTP 500 response.
We should verify, but I think that the load balancers give up after 30 seconds of inactivity, so as long as data is flowing, things are fine.

Flask does make streaming pretty easy, thankfully:
However, the mapfile data is sorted, so streaming may not reduce time-to-first-byte.
(In reply to Hal Wine [:hwine] (use needinfo) from comment #2)
> However, the mapfile data is sorted, so streaming may not reduce
> time-to-first-byte.

There is a delay from querying the DB until starting to receive rows back via the database connection, and there is a second delay from receiving the first data back from the database connection until the full mapfile is built up in memory. I think we can impact this second delay with streaming, but you are quite right Hal that the first part also incurs a delay. I'm not sure how much is the first part, and how much is the second part, at the moment.
Whiteboard: [kanban:engops:]
Whiteboard: [kanban:engops:] → [kanban:engops:]
Assignee: nobody → hwine
no one to work on this at the moment

If anyone is seeing this in the real world, could they add details to help with prioritization.
Assignee: hwine → nobody
No need to touch this. No problems in over a year of use, and not a system we need to heavily invest in, especially with deprioritisation of B2G which was main user.
Closed: 5 years ago
Resolution: --- → WONTFIX
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.