1563230 - cronrun command can hang

Status: RESOLVED FIXED

(Socorro :: General, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Philipp noticed archivescraper hadn't run and picked up 68.0rc1 and wrote up bug #1563172.

The last cron_log entry is this one:

July 1st, 8:38 pm
archivescraper
<class 'concurrent.futures.process.BrokenProcessPool'> BrokenProcessPool('A process in the process pool was terminated abruptly while the future was running or pending.',)

Brian looked at the node and the Docker container is running this:

[bpitts@blahblahblah ~]$ sudo .local/bin/py-spy --nonblocking --dump --pid 17504
/usr/local/bin/python3.6
Python version 3.6.8
Thread 0x7FB21FE0C700 (active)
     _send (connection.py:368)
     _send_bytes (connection.py:398)
     send_bytes (connection.py:200)
     _feed (queues.py:240)
     run (threading.py:864)
     run (threading.py:45)
     _bootstrap_inner (threading.py:916)
     _bootstrap (threading.py:884)
Thread 0x7FB22F97E400 (active)
     _wait_for_tstate_lock (threading.py:1072)
     join (threading.py:1056)
     _finalize_join (queues.py:191)
     __call__ (util.py:186)
     _run_finalizers (util.py:262)
     _exit_function (util.py:322)

We haven't seen another case of cronrun hanging as far as I know.

We should wrap cronrun with timeout much like we do with minidump-stackwalk.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P2 to do soon. Having cronrun get sad isn't great.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4990: bug 1563230: wrap cronrun in a timeout

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #4990: "bug 1563230: wrap cronrun in a timeout" in 119b6e9.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is in prod now. Marking as FIXED.