1199347 - SETA stopped working for 6 hrs on Aug 27, then started working again, why?

Status: RESOLVED DUPLICATE of bug 1176784

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

From bug  1199226 

So I looked at the logs on the scheduling master
I looks like SETA stopped skipping jobs from 00:51 until 06:37

I just grepped for "skipped" for today's logs on bm81

2015-08-27 00:51:07-0700 [-] tests-fx-team-ubuntu64_vm-opt-unittest-7-3600: skipping with 2/7 important changes since only 151/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-fx-team-yosemite-debug-unittest-7-3600: skipping with 1/7 important changes since only 229/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-mozilla-inbound-win7-ix-debug-unittest-7-3600: skipping with 1/7 important changes since only 3570/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-mozilla-inbound-yosemite-opt-unittest-7-3600: skipping with 1/7 important changes since only 467/3600s have elapsed
2015-08-27 06:37:43-0700 [-] tests-fx-team-ubuntu64_vm_mobile-opt-unittest-7-3600: skipping with 4/7 important changes since only 277/3600s have elapsed
2015-08-27 06:38:05-0700 [-] tests-fx-team-ubuntu64_vm_armv7_large-opt-unittest-7-3600: skipping with 4/7 important changes since only 47/3600s have elapsed

This same problem didn't occur the 26th, I looked at the logs.

I still don't understand why SETA skipping jobs stopped at 00:51.  I also don't understand why it started skipping jobs again at 6:37am.  Will investigate.

However, probably the machine pools were able to keep up with load overnight but when daytime ET hit, the number of pushes increased and caused the high pending counts w/o SETA.  

Looking at m-i there certainly was reason to skip jobs from midnight to 6am giving the intervals of pushes.

Comment 2

[Kim Moir [:kmoir] ET]

Also, I wonder if there is something weird happening in bb because our masters have been up so long. They were due to be rebooted weeks ago but this has been deferred to the TCW Sat aug 29

Comment 3

[Kim Moir [:kmoir] ET]

I suspect bug 1176784 is the cause.