Bugzilla

(In reply to Ionuț Goldan [:igoldan] from comment #10)
> I'm not really sure if this is a problem of missing jobs. I think this has more to do with why we're getting alerts on such old data points.
> In general, alerts like these are generated when sheriffs trigger data points from old pushes. But we didn't see any such activity.

Is it possible someone accidentally triggered alerts on these old pushes (maybe while thinking they were testing changes on stage)?

> This feels to me like something happened with the ingestion pipeline *and/or* with the data from that time interval *(Oct 10 up to Nov 10)*.
> During that time, we had some very serious changes, such as meta bug 1597476 or the deployment of the new Taskcluster *(forgot when that happened)*.
> 
> The meta bug came with changes to Perfherder's pipeline & other manual interventions over the Celery queues *(if I'm correct)*. Regarding the Taskcluster deploy... Don't know what implications that could have brought.

That taskcluster changeover was on Nov 9th. How would the ingestion or API changes cause pushes during that time to have alerts generated for a future date though? I'm not at all familiar with how that alert logic works.

(In reply to Ionuț Goldan [:igoldan] from comment #10)
> I'm not really sure if this is a problem of missing jobs. I think this has more to do with why we're getting alerts on such old data points.
> In general, alerts like these are generated when sheriffs trigger data points from old pushes. But we didn't see any such activity.

Is it possible someone accidentally triggered alerts on these old pushes (maybe while thinking they were testing changes on stage)?

> This feels to me like something happened with the ingestion pipeline *and/or* with the data from that time interval *(Oct 10 up to Nov 10)*.
> During that time, we had some very serious changes, such as meta bug 1597476 or the deployment of the new Taskcluster *(forgot when that happened)*.
> 
> The meta bug came with changes to Perfherder's pipeline & other manual interventions over the Celery queues *(if I'm correct)*. Regarding the Taskcluster deploy... Don't know what implications that could have brought.

That taskcluster changeover was on Nov 9th. How would the ingestion or API changes cause pushes during that time to have alerts generated for a future date though (and only for those during that time period)? I'm not familiar with how all of the alert logic works.

(In reply to Ionuț Goldan [:igoldan] from comment #10)
> I'm not really sure if this is a problem of missing jobs. I think this has more to do with why we're getting alerts on such old data points.
> In general, alerts like these are generated when sheriffs trigger data points from old pushes. But we didn't see any such activity.

Is it possible someone accidentally triggered alerts on these old pushes (maybe while thinking they were testing changes on stage)?

> This feels to me like something happened with the ingestion pipeline *and/or* with the data from that time interval *(Oct 10 up to Nov 10)*.
> During that time, we had some very serious changes, such as meta bug 1597476 or the deployment of the new Taskcluster *(forgot when that happened)*.
> 
> The meta bug came with changes to Perfherder's pipeline & other manual interventions over the Celery queues *(if I'm correct)*. Regarding the Taskcluster deploy... Don't know what implications that could have brought.

That taskcluster changeover was on Nov 9th. How would the ingestion or API changes cause pushes during that time to have alerts generated for a future date though (and only for those during that time period)? I'm not familiar with how all of the alert logic works but yes, this will require a thorough investigation.