Closed Bug 1501386 Opened 7 years ago Closed 6 years ago

validate_signature lambda fails

Categories

(Cloud Services :: Operations: Kinto, task)

Product:
Cloud Services
Component:
Operations: Kinto
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wezhou, Assigned: leplatrem)

Details

validate_signature lambda in -prod started to fail at 21:55 UTC 2018/10/22 with execution timeout errors. Before the failure started to happen, at 21:47 UTC there happened to be some updates to the personality-provider-models collection, the timeout error may be related to those since the timing of the two happening is very close. The execution timeout error last about 18 hours, till 15:20 UTC 10/23, and then the error changed to "Process exited before completing request", for example, Read collection list from /buckets/monitor/collections/changes 01/35 /buckets/main/collections/tippytop: OK 02/35 /buckets/main-preview/collections/tippytop: OK END RequestId: fca62f94-d6ee-11e8-8e5f-95e464bdba6a REPORT RequestId: fca62f94-d6ee-11e8-8e5f-95e464bdba6a Duration: 54741.36 ms Billed Duration: 54800 ms Memory Size: 128 MB Max Memory Used: 128 MB RequestId: fca62f94-d6ee-11e8-8e5f-95e464bdba6a Process exited before completing request This error has replaced the previous execution timeout error, and has been happening since.
Assignee: nobody → mathieu
I made some changes in the lambda to make it CPU bound. https://github.com/mozilla-services/amo2kinto-lambda/releases/tag/4.4.0 On my machine, it's super fast, so really, I don't get how it could take 18hours (??)
The lambda runs every 5 minutes. What I meant was that for that 18 hours, every run failed with the "execution timeout" error. Anyways, I think both errors (the execution timeout and the process-exited-before-completing-request) may have to do with limited memory resource. Previously we didn't have too large collections and 128 MB was more than enough for the lambda to finish its job. We have increased -prod memory size to 1024 MB, and the lambda is now able to finish successfully.

This hasn't happened for a while since we increased the lambda's memory size. Close it for now.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.