Because the only things that are still running on linux64 buildbot slaves that we watch_pending for are Thunderbird and mozilla-release/esr (and mozilla-beta add-on-devel builds once bug 1200346 lands), on a Sunday the periodic file updates can run 5 hours late because they have to wait for idle slaves coming from Thunderbird nightlies.
Created attachment 8855828 [details] periodic.PNG So by "watch_pending" we are referring to the pending jobs from buildapi (https://secure.pub.build.mozilla.org/buildapi/pending)? We have "Linux x86-64 mozilla-aurora periodic file update" builders with Unknown revision,example in the periodic.png attachment.
watch_pending as in https://github.com/mozilla-releng/build-cloud-tools/blob/master/configs/watch_pending.cfg which is what causes us to *not* have them hanging out in buildapi/pending, because it sees that they are pending and that we need to start up some AWS instances to run them.
Created attachment 8856544 [details] Adding periodic update to watch_pending.cfg
Comment on attachment 8856544 [details] Adding periodic update to watch_pending.cfg LGTM!
Comment on attachment 8856544 [details] Adding periodic update to watch_pending.cfg https://github.com/mozilla-releng/build-cloud-tools/commit/30089d7ca2c96e25ae36f47964b8c2528cf08552