Closed Bug 1037874 Opened 10 years ago Closed 10 years ago

add timeout to fetch-adi-from-hive script

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: rhelmer)

References

Details

In bug 1037870 we saw the fetch-adi-from-hive job hang indefinitely reading from Hive. We need to add a timeout to this script (we've had similar issues with ftpscraper and used this solution)
Blocks: 1037870
No longer blocks: 1037870
See Also: → 1037870
the pyhs2 library that implements the connection to Hive doesn't support timeout. However, that library uses the same Thrift implementation as our older HBase library. We added timeout back in '10 to that library, we could offer a patch upstream to the maintainer of pyhs2...
someone has already submitted such a patch: https://github.com/BradRuderman/pyhs2/pull/20 we just need to encourage the maintainer to accept it...
Commit pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/309f393b6edbe751ee0a03f0f4636f1a1a197635 Merge pull request #2215 from twobraids/pyhs2-with-timeout Fixes Bug 1037874 - pyhs2 with timeout
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
This seems to be timing out after less than a second :) So either we're misinterpreting the config value, or it's not working quite right...
Status: RESOLVED → REOPENED
Flags: needinfo?(lars)
Resolution: FIXED → ---
OK peterbe and I investigated - we're passing seconds while TSocket.setTimeout() wants ms: http://people.apache.org/~thejas/thrift-0.9/javadoc/org/apache/thrift/transport/TSocket.html#setTimeout%28int%29 (I am having trouble finding docs for the python thrift bindings online, but it's generated from the same source as the Java ones ^)
Flags: needinfo?(lars)
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/90bf9fb18cb51a56542229125fe78b8bebb03e68 fix bug 1037874 - pyhs2 TSocket.setTimeout() wants ms not seconds https://github.com/mozilla/socorro/commit/755e8d260ed6c8428ab77e7ba8ef7458958dac5b Merge pull request #2233 from rhelmer/bug1037874-pyhs2-timeout-ms fix bug 1037874 - pyhs2 TSocket.setTimeout() wants ms not seconds
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → 95
You need to log in before you can comment on or make changes to this bug.