The issues arises after a reconfig which changes the list of builders and it clears once the Heroku's dynos restart the next day. This will come out of a fix in mozci: https://github.com/mozilla/mozilla_ci_tools/issues/365
Let's assume for automatic backfilling that allthethings.json will always be out of date. It gets generated every 15 minutes on the clock and the new file becomes public once an hour on the 10th minute. That means that automatic backfilling will start hearing about new builders before we can actually determine if they're valid or not. If we hear about a builder, we will *not* verify it and carry on. This might be an issue with a builder being removed by the time we try to schedule it. In that case, we should recover by catching the error from Buildapi.
I believe this is not anymore an issue after disabling verifications. https://github.com/mozilla/pulse_actions/blob/master/pulse_actions/worker.py#L191