1152294 - Bugzilla Elasticsearch Cluster is Stale (has ETL has stopped?)

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

The public and private clusters have stopped receiving updates from the ETL processes.  The last update was done around Mar 29 @ 14:46EDT.

Please check that the ETL processes, both public and private, are running. I may require logs if they are.

[1] Diagram with bug numbers:  https://github.com/klahnakoski/Bugzilla-ETL/blob/master/docs/Architecture%20%28bug%20879822%29.png

Comment 1

[Kendall Libby [:fubar] (he/him)]

There was a hung private process from March 29; killed it and the next cron job ran and did stuff. The public one looks like it's been running fine, from the logs.

Comment 2

[Kendall Libby [:fubar] (he/him)]

And now it looks like the private one may be wedged again?  It's still running, and the logs are showing:

2015-04-08 13:24:49.233966 - Waiting on thread "etl"

2015-04-08 13:24:56.555011 - Waiting on thread "etl"

2015-04-08 13:25:10.284315 - Waiting on thread "etl"

strace doesn't help much:
[pid 23430] futex(0x3113020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 23482] futex(0x3113020, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 23430] <... futex resumed> )       = 0
[pid 23430] futex(0x3113020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 23482] futex(0x3113020, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 23430] <... futex resumed> )       = 0
[pid 23482] futex(0x3113020, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 23430] futex(0x3113020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 23482] <... futex resumed> )       = 0
[pid 23430] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 23430] futex(0x3113020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 23482] futex(0x3113020, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 23430] <... futex resumed> )       = 0
[pid 23430] futex(0x3113020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 23482] futex(0x3113020, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 23430] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 23482] <... futex resumed> )       = 0

Comment 3

[Kyle Lahnakoski [:ekyle]]

The ETL should be generating a log file (exact location in setting.json file)  which may give more infor to what's happening (problem connecting?)

Comment 4

[Kyle Lahnakoski [:ekyle]]

The "Waiting on thread \"etl\"" without logs between means the etl thread is slow (or failed).   Again the log file should tell us more.

Comment 5

[Kyle Lahnakoski [:ekyle]]

The ETL appears to have caught up fine.  This bug will be kept open until fubar and I are happy it will not show another problem.

Comment 6

[Kyle Lahnakoski [:ekyle]]

Looks good!