Closed
Bug 785531
Opened 12 years ago
Closed 12 years ago
Validation timeouts
Categories
(addons.mozilla.org Graveyard :: Add-on Validation, defect)
addons.mozilla.org Graveyard
Add-on Validation
Tracking
(Not tracked)
VERIFIED
FIXED
2012-08-30
People
(Reporter: krupa.mozbugs, Unassigned)
References
()
Details
We are seeing a lot of Validation timeouts in AMO Prod. See https://addons.mozilla.org/en-US/developers/addon/test-pilot/validation-result/131205
and screenshot at http://cl.ly/image/2k1l2q153g2a
Comment 1•12 years ago
|
||
Jason, is celery looking ok?
Comment 2•12 years ago
|
||
I ran one of the add-ons locally and it's only taking ~12s to validate. I don't think there have been any significant changes to the validator that would prompt these timeouts.
Comment 3•12 years ago
|
||
I ran the compatibility bump today, so that's probably the reason for the slower validations and timeouts. I saw a few timoeout errors in the failing list of add-ons.
Comment 4•12 years ago
|
||
The concept of a work queue is that the server always does the same amount of work, it might just take longer to pick up a job. If compatibility bumps are causing timeouts then that's a serious problem; we may have to adjust some queue settings.
Comment 5•12 years ago
|
||
(In reply to Kumar McMillan [:kumar] from comment #1)
> Jason, is celery looking ok?
The queue looks okay now, but the bulk queue alerted earlier today around the same time the validation errors occurred:
Fri 10:04:59 PDT [171] celery1.addons.phx1.mozilla.com:AMO Celery - RabbitMQ Queue prod_zamboni bulk is CRITICAL: CRITICAL: 4617 unacknowledged messages in queue bulk on vhost prod_zamboni.
Fri 10:25:00 PDT [174] celery1.addons.phx1.mozilla.com:AMO Celery - RabbitMQ Queue prod_zamboni bulk is OK: OK: 0 unacknowledged messages in queue bulk on vhost prod_zamboni.
Comment 6•12 years ago
|
||
ah, ok. The bulk queue gets a sudden burst of jobs and then will work out those jobs at its own pace. Putting an alert on a large number of unacknowledged messages might be a false alarm -- it is expected. However, we might want to revisit the settings for how many jobs run concurrently in the bulk queue so that it's not putting too much stress on the cluster.
Comment 7•12 years ago
|
||
Jorge Villalobos asked me to add a comment to this bug as I was personally seeing this when I tried to upload a new version of my Web Developer extension yesterday. Here is a screenshot of the error I saw:
http://stuff.chrispederick.com/J1Ai
The new version of the extension is a little bigger than the previous release (about 1.34MB) but I'm not sure if that is causing any issues. I can attach the XPI if that is useful - just let me know.
Comment 8•12 years ago
|
||
Just a quick follow up to say that I just tried again this morning and I'm still seeing the same problem.
Comment 9•12 years ago
|
||
Jason: what is the load like on these boxes? It sounds like we need to back off on the queue settings
Comment 10•12 years ago
|
||
The load looks okay overall, but does spike when there are jobs in the rabbit queue [1]. I reduced the bulk workers from 12 -> 6 for now.
[1]https://ganglia-phx1.mozilla.org/ganglia/?r=week&cs=&ce=&c=Addons+Celery&h=celery1&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS
Comment 11•12 years ago
|
||
Bug 786292 to add additional celery node to the prod cluster.
Comment 12•12 years ago
|
||
I haven't seen these since the new node. Krupa?
Comment 13•12 years ago
|
||
I can confirm that I was just able to upload my new version of the Web Developer extension so this looks to be resolved to me.
Comment 14•12 years ago
|
||
thanks everyone
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment 15•12 years ago
|
||
I just performed many validations without any problem. Closing bug.
Status: RESOLVED → VERIFIED
Comment 16•12 years ago
|
||
It seems this bug is not fixed. I still receive validation timeouts. There is no indication what cause the long validation. Too many files? To much javascript code validation? ... ? Why, oh why became developing addons for Firefox such a painful experience over the last years?
Assignee | ||
Updated•9 years ago
|
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•