Just stumbled upon on a task which i supposed to be indexed, but it's not: https://tools.taskcluster.net/task-inspector/#ZQRDYb1gTFWS2Isg4ZgSEQ/ I see the index routes set, but I can't find this task in the index.
Hmm, another one: https://tools.taskcluster.net/task-inspector/#NTtZfqgRTlumGSU2uU4oIw/
Any idea what may be going on here? This is something new, I have never seen this until today...
Indeed the update seems to have failed. I compared task definitions between: a) https://queue.taskcluster.net/v1/task/M-RG5Y_6TreUOC_O4LUhbA b) https://queue.taskcluster.net/v1/task/ZQRDYb1gTFWS2Isg4ZgSEQ and there are not any obvious differences. a) was able to update the index but b) was not. I couldn't see anything obvious in papertrail to give an indication of a problem. Also, status.taskcluster.net says the index has been up since August last year without outage (at least none that it detected that lasted > 5 mins or so). At this point, the most likely candidate to me would be either a pulse outage, or connectivity issues. Task a) above was successful, which ran only a few hours before task b) that didn't update the index, so I don't think this is related to the recent pulse certificate problem we had a few days ago. @Mark, Jonas: are you aware of any pulse outages / network disturbances, or have other ideas about what might have caused this index update failure? @Jonas, does the index log the pulse messages it receives/processes? The task in question was created at 2016-02-23T23:06:11.791Z. Pete
Found another one: https://tools.taskcluster.net/task-inspector/#OCi3hgBmQY6lAbl-2dW_mw/
Got a warning from pulse guardian appears our indexing process is down I suppose Bstack rolled out updates yesterday... We will have a look and fix this ASAP.
bstack is working to fix, we rolled out refactoring yesterday... Some how the background process ran the server code instead :) So a background process was running, just not the right one. Good news, messages seems to still be in the pulse queue so we'll catch up soon: queue/taskcluster-index/index/incoming-tasks 8369 messages So credits to mcote for keeping pulse alive :)
Woot! Thanks a lot.
All messages are now processed... index is up to date... Sorry, we took a bit longer than usual to index tasks. I'm suspect this had negative effects for caches. But I'm really happy it proved robust and was able to process the message backlog :)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Sweet! I've got my indexes now. TYVM.
You need to log in before you can comment on or make changes to this bug.