Status

Taskcluster
Index
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rail, Unassigned)

Tracking

Details

(Reporter)

Description

2 years ago
Just stumbled upon on a task which i supposed to be indexed, but it's not:
https://tools.taskcluster.net/task-inspector/#ZQRDYb1gTFWS2Isg4ZgSEQ/

I see the index routes set, but I can't find this task in the index.
(Reporter)

Updated

2 years ago
Blocks: 1118794
(Reporter)

Comment 2

2 years ago
Any idea what may be going on here? This is something new, I have never seen this until today...
Indeed the update seems to have failed. I compared task definitions between:

a) https://queue.taskcluster.net/v1/task/M-RG5Y_6TreUOC_O4LUhbA
b) https://queue.taskcluster.net/v1/task/ZQRDYb1gTFWS2Isg4ZgSEQ

and there are not any obvious differences. a) was able to update the index but b) was not.

I couldn't see anything obvious in papertrail to give an indication of a problem.

Also, status.taskcluster.net says the index has been up since August last year without outage (at least none that it detected that lasted > 5 mins or so).

At this point, the most likely candidate to me would be either a pulse outage, or connectivity issues.

Task a) above was successful, which ran only a few hours before task b) that didn't update the index, so I don't think this is related to the recent pulse certificate problem we had a few days ago.

@Mark, Jonas: are you aware of any pulse outages / network disturbances, or have other ideas about what might have caused this index update failure?

@Jonas, does the index log the pulse messages it receives/processes?

The task in question was created at 2016-02-23T23:06:11.791Z.

Pete
Flags: needinfo?(mcote)
Flags: needinfo?(jopsen)
Got a warning from pulse guardian appears our indexing process is down I suppose

Bstack rolled out updates yesterday... We will have a look and fix this ASAP.
Flags: needinfo?(jopsen)
bstack is working to fix, we rolled out refactoring yesterday...
Some how the background process ran the server code instead :)

So a background process was running, just not the right one.


Good news, messages seems to still be in the pulse queue so we'll catch up soon:
  queue/taskcluster-index/index/incoming-tasks 8369 messages

So credits to mcote for keeping pulse alive :)
Flags: needinfo?(mcote)
(Reporter)

Comment 7

2 years ago
Woot! Thanks a lot.
All messages are now processed... index is up to date... Sorry, we took a bit longer than usual to index tasks.

I'm suspect this had negative effects for caches. But I'm really happy it proved robust and was able to process the message backlog :)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 9

2 years ago
Sweet! I've got my indexes now. TYVM.
You need to log in before you can comment on or make changes to this bug.