Validation timeouts

VERIFIED FIXED in 2012-08-30


6 years ago
3 years ago


(Reporter: krupa.mozbugs, Unassigned)







6 years ago
We are seeing a lot of Validation timeouts in AMO Prod. See

and screenshot at
Jason, is celery looking ok?
I ran one of the add-ons locally and it's only taking ~12s to validate. I don't think there have been any significant changes to the validator that would prompt these timeouts.
I ran the compatibility bump today, so that's probably the reason for the slower validations and timeouts. I saw a few timoeout errors in the failing list of add-ons.
The concept of a work queue is that the server always does the same amount of work, it might just take longer to pick up a job. If compatibility bumps are causing timeouts then that's a serious problem; we may have to adjust some queue settings.
(In reply to Kumar McMillan [:kumar] from comment #1)
> Jason, is celery looking ok?

The queue looks okay now, but the bulk queue alerted earlier today around the same time the validation errors occurred: 

Fri 10:04:59 PDT [171] Celery - RabbitMQ Queue prod_zamboni bulk is CRITICAL: CRITICAL: 4617 unacknowledged messages in queue bulk on vhost prod_zamboni. 
Fri 10:25:00 PDT [174] Celery - RabbitMQ Queue prod_zamboni bulk is OK: OK: 0 unacknowledged messages  in queue bulk on vhost prod_zamboni.
ah, ok. The bulk queue gets a sudden burst of jobs and then will work out those jobs at its own pace. Putting an alert on a large number of unacknowledged messages might be a false alarm -- it is expected. However, we might want to revisit the settings for how many jobs run concurrently in the bulk queue so that it's not putting too much stress on the cluster.

Comment 7

6 years ago
Jorge Villalobos asked me to add a comment to this bug as I was personally seeing this when I tried to upload a new version of my Web Developer extension yesterday. Here is a screenshot of the error I saw:

The new version of the extension is a little bigger than the previous release (about 1.34MB) but I'm not sure if that is causing any issues. I can attach the XPI if that is useful - just let me know.

Comment 8

6 years ago
Just a quick follow up to say that I just tried again this morning and I'm still seeing the same problem.
Jason: what is the load like on these boxes?  It sounds like we need to back off on the queue settings
The load looks okay overall, but does spike when there are jobs in the rabbit queue [1]. I reduced the bulk workers from 12 -> 6 for now.

Bug 786292 to add additional celery node to the prod cluster.
I haven't seen these since the new node.  Krupa?

Comment 13

6 years ago
I can confirm that I was just able to upload my new version of the Web Developer extension so this looks to be resolved to me.
thanks everyone
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 15

6 years ago
I just performed many validations without any problem. Closing bug.

Comment 16

6 years ago
It seems this bug is not fixed. I still receive validation timeouts. There is no indication what cause the long validation. Too many files? To much javascript code validation? ... ? Why, oh why became developing addons for Firefox such a painful experience over the last years?
Product: → Graveyard
You need to log in before you can comment on or make changes to this bug.