Validation queues repeatedly back up

RESOLVED FIXED in 2015-06

Status

RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: kmag, Assigned: kmag)

Tracking

unspecified
2015-06

Details

(Whiteboard: [june-launch][qa-])

(Assignee)

Description

4 years ago
+++ This bug was initially created as a clone of Bug #1135471 +++

Since this seems to be a recurring problem, I'm opening a separate bug to prevent it from happening again.

I know Jason mentioned at one point that the tasks don't have a hard time limit, so maybe the fix is to add one.

Please feel free to close if something has already been done about this.

Thanks
(Assignee)

Updated

4 years ago
Assignee: nobody → kmaglione+bmo
Whiteboard: [june-launch]
(Assignee)

Comment 1

4 years ago
The problem seems to be that the validator is implementing a timeout by installing a SIGALRM handler and setting a timeout, without unsetting that timeout or removing its handler when validation is complete or the task fails.

So I think this means that when validation does not complete successfully, the timer callback will raise an exception at an arbitrary time during the worker process's execution.
(Assignee)

Comment 4

4 years ago
Huh. Those PR numbers are really conveniently symmetrical.
(Assignee)

Updated

4 years ago
Target Milestone: --- → 2015-06
(Assignee)

Updated

4 years ago
Whiteboard: [june-launch] → [june-launch][qa-]

Comment 5

4 years ago
Commits pushed to master at https://github.com/mozilla/amo-validator

https://github.com/mozilla/amo-validator/commit/d75eee3108a3099c426fba5da7d2aa8ec046e0db
Bug 1163799: Default to no timeout except when run from the command line, and clean up timers and SIGALRM handlers on completion.

https://github.com/mozilla/amo-validator/commit/347191aa06f8a897e42ebbc24f9edbba99186b2b
Merge pull request #300 from kmaglione/bug-1163799

Bug 1163799: Fix issues with the celery queues when validator tasks fail

Comment 6

4 years ago
Commits pushed to master at https://github.com/mozilla/olympia

https://github.com/mozilla/olympia/commit/8dd9db7fa977f5edca5164772ae7508337912222
Bug 1163799: Use Celery soft timeouts rather than the validator's problematic setitimer/SIGALRM implementation.

https://github.com/mozilla/olympia/commit/6f7d9fe372395bfabd3085beaaeb54f89c64fc6a
Merge pull request #600 from kmaglione/bug-1163799

Bug 1163799: Use Celery soft timeouts rather than the validator's pro…
(Assignee)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.