It looks like celery's task dispatcher is introducing some recursion and thus, over time, we are running out of stacks (Python's limit is 1000). See bug 735473 As a quick fix, perhaps we can just limit the number of tasks a validator worker executes before it is replaced by a new process. We can set: CELERYD_MAX_TASKS_PER_CHILD or use --maxtasksperchild when starting the worker http://celery.readthedocs.org/en/latest/userguide/workers.html#max-tasks-per-child-setting
Added --maxtasksperchild=50 to devhub workers. Hopefully this will help.