594453 - Socorro - collector reimplementation with timeout

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[K Lars Lohn [:lars] [:klohn]]

The collector makes http requests to processors to queue jobs for processing.  If there are problems with that connection, there could be long waits for timeouts.  This could result in crash submissions getting dropped because the Apache is too busy waiting to service them.  The collector needs a timeout.

There are two ways to implement this:

1) add a global socket timeout.  This will prevent the http request blocking too long.  It could effect other socket connections like thrift, and if set shorter than the thrift timeout, could override that setting.

2) reimplement the http request to use httplib instead of urllib2.  That module has a built in timeout feature with, supposedly, won't interfere with any timeouts set in thrift.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

completed. the urllib2 did actually add a timeout parameter.  I was looking at outdated documentation.

there is a new config parameter in the collector called 'processorSubmissionTimeout'

Comment 2

[Laura Thomson :laura]

Not applicable to 1.7.x