1809500 - consider dropping gevent complexity

Status: RESOLVED FIXED

(Socorro :: Antenna, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Antenna (crash ingestion pipeline collector) tries very hard to accept a crash report, throttle it, return an HTTP response, and end the submission HTTP request as quickly as possible. One of the things it does is put the crash report parts in a work queue for a crash mover coroutine to pick up and then upload to AWS S3 later--outside of the HTTP request/response cycle.

Antenna has been architected this way for a long while. Antenna emits a metric for the total amount of time between starting handling of the HTTP request to when the crash mover coroutine is done its work. Looking at data over the past 3 months, this is almost always < 5s.

Given that, I think it's time we change how things work and move uploading of the crash report bits to AWS S3 into the HTTP request/response cycle and ditch all the complexity around the work queue and crash mover coroutines.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this block the GCP migration because I had issues with Google client Python libraries and gevent in the past.

Comment 2

[Daniel Thorn [:relud]]

Attachment: [mozilla-services/antenna] bug-1809500: make crashmover run synchronously (#963)

Comment 3

[Daniel Thorn [:relud]]

relud merged PR [mozilla-services/antenna]: bug-1809500: make crashmover run synchronously (#963) in 8107c88.

Comment 4

[Daniel Thorn [:relud]]

Attachment: [mozilla-services/antenna] bug-1809500: use gevent.sleep for crashmover retry (#970)

Comment 5

[Daniel Thorn [:relud]]

relud closed PR [mozilla-services/antenna]: bug-1809500: use gevent.sleep for crashmover retry (#970) in #970 (comment).

Comment 6

[Daniel Thorn [:relud]]

Attachment: [mozilla-services/antenna] bug-1809500: drop gevent and switch gunicorn to default worker (#971)

Comment 7

[Daniel Thorn [:relud]]

relud merged PR [mozilla-services/antenna]: bug-1809500: drop gevent and switch gunicorn to default worker (#971) in 781af34.

Comment 8

[Daniel Thorn [:relud]]

Attachment: [mozilla-services/cloudops-deployment] bug-1809500: Use gunicorn sync worker in socorro collector (#4928)

Comment 9

[Daniel Thorn [:relud]]

willkg merged PR [mozilla-services/cloudops-deployment]: bug-1809500: Use gunicorn sync worker in socorro collector (#4928) in 42078da.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

This went out in bug #1885986. POST handling times mean when from 8ms to 44ms and (95 percentile) went from like 50ms to 250ms. That's expected since it's doing more work, but the numbers are fine. Marking as FIXED.