Closed Bug 1163799 Opened 9 years ago Closed 9 years ago

Validation queues repeatedly back up

Categories

(addons.mozilla.org Graveyard :: Developer Pages, defect)

Product:
addons.mozilla.org Graveyard
Component:
Developer Pages
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
2015-06

People

(Reporter: kmag, Assigned: kmag)

References

Details

(Whiteboard: [june-launch][qa-]) 

+++ This bug was initially created as a clone of Bug #1135471 +++

Since this seems to be a recurring problem, I'm opening a separate bug to prevent it from happening again.

I know Jason mentioned at one point that the tasks don't have a hard time limit, so maybe the fix is to add one.

Please feel free to close if something has already been done about this.

Thanks
Assignee: nobody → kmaglione+bmo
Whiteboard: [june-launch]
The problem seems to be that the validator is implementing a timeout by installing a SIGALRM handler and setting a timeout, without unsetting that timeout or removing its handler when validation is complete or the task fails.

So I think this means that when validation does not complete successfully, the timer callback will raise an exception at an arbitrary time during the worker process's execution.
Huh. Those PR numbers are really conveniently symmetrical.
Target Milestone: --- → 2015-06
Whiteboard: [june-launch] → [june-launch][qa-]
Commits pushed to master at https://github.com/mozilla/amo-validator

https://github.com/mozilla/amo-validator/commit/d75eee3108a3099c426fba5da7d2aa8ec046e0db
Bug 1163799: Default to no timeout except when run from the command line, and clean up timers and SIGALRM handlers on completion.

https://github.com/mozilla/amo-validator/commit/347191aa06f8a897e42ebbc24f9edbba99186b2b
Merge pull request #300 from kmaglione/bug-1163799

Bug 1163799: Fix issues with the celery queues when validator tasks fail
Commits pushed to master at https://github.com/mozilla/olympia

https://github.com/mozilla/olympia/commit/8dd9db7fa977f5edca5164772ae7508337912222
Bug 1163799: Use Celery soft timeouts rather than the validator's problematic setitimer/SIGALRM implementation.

https://github.com/mozilla/olympia/commit/6f7d9fe372395bfabd3085beaaeb54f89c64fc6a
Merge pull request #600 from kmaglione/bug-1163799

Bug 1163799: Use Celery soft timeouts rather than the validator's pro…
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.