1250720 - Task not indexed

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Rail Aliiev [:rail]]

Just stumbled upon on a task which i supposed to be indexed, but it's not:
https://tools.taskcluster.net/task-inspector/#ZQRDYb1gTFWS2Isg4ZgSEQ/

I see the index routes set, but I can't find this task in the index.

Comment 1

[Rail Aliiev [:rail]]

Hmm, another one:

https://tools.taskcluster.net/task-inspector/#NTtZfqgRTlumGSU2uU4oIw/

Comment 2

[Rail Aliiev [:rail]]

Any idea what may be going on here? This is something new, I have never seen this until today...

Comment 3

[Pete Moore [:pmoore][:pete]]

Indeed the update seems to have failed. I compared task definitions between:

a) https://queue.taskcluster.net/v1/task/M-RG5Y_6TreUOC_O4LUhbA
b) https://queue.taskcluster.net/v1/task/ZQRDYb1gTFWS2Isg4ZgSEQ

and there are not any obvious differences. a) was able to update the index but b) was not.

I couldn't see anything obvious in papertrail to give an indication of a problem.

Also, status.taskcluster.net says the index has been up since August last year without outage (at least none that it detected that lasted > 5 mins or so).

At this point, the most likely candidate to me would be either a pulse outage, or connectivity issues.

Task a) above was successful, which ran only a few hours before task b) that didn't update the index, so I don't think this is related to the recent pulse certificate problem we had a few days ago.

@Mark, Jonas: are you aware of any pulse outages / network disturbances, or have other ideas about what might have caused this index update failure?

@Jonas, does the index log the pulse messages it receives/processes?

The task in question was created at 2016-02-23T23:06:11.791Z.

Pete

Comment 4

[Rail Aliiev [:rail]]

Found another one:
https://tools.taskcluster.net/task-inspector/#OCi3hgBmQY6lAbl-2dW_mw/

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

Got a warning from pulse guardian appears our indexing process is down I suppose

Bstack rolled out updates yesterday... We will have a look and fix this ASAP.

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

bstack is working to fix, we rolled out refactoring yesterday...
Some how the background process ran the server code instead :)

So a background process was running, just not the right one.


Good news, messages seems to still be in the pulse queue so we'll catch up soon:
  queue/taskcluster-index/index/incoming-tasks 8369 messages

So credits to mcote for keeping pulse alive :)

Comment 7

[Rail Aliiev [:rail]]

Woot! Thanks a lot.

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

All messages are now processed... index is up to date... Sorry, we took a bit longer than usual to index tasks.

I'm suspect this had negative effects for caches. But I'm really happy it proved robust and was able to process the message backlog :)

Comment 9

[Rail Aliiev [:rail]]

Sweet! I've got my indexes now. TYVM.