SETA stopped working for 6 hrs on Aug 27, then started working again, why?

RESOLVED DUPLICATE of bug 1176784

Status

Infrastructure & Operations
Buildduty
RESOLVED DUPLICATE of bug 1176784
3 years ago
16 days ago

People

(Reporter: kmoir, Assigned: kmoir)

Tracking

Details

(URL)

(Assignee)

Description

3 years ago
From bug  1199226 

So I looked at the logs on the scheduling master
I looks like SETA stopped skipping jobs from 00:51 until 06:37

I just grepped for "skipped" for today's logs on bm81

2015-08-27 00:51:07-0700 [-] tests-fx-team-ubuntu64_vm-opt-unittest-7-3600: skipping with 2/7 important changes since only 151/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-fx-team-yosemite-debug-unittest-7-3600: skipping with 1/7 important changes since only 229/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-mozilla-inbound-win7-ix-debug-unittest-7-3600: skipping with 1/7 important changes since only 3570/3600s have elapsed
2015-08-27 00:51:07-0700 [-] tests-mozilla-inbound-yosemite-opt-unittest-7-3600: skipping with 1/7 important changes since only 467/3600s have elapsed
2015-08-27 06:37:43-0700 [-] tests-fx-team-ubuntu64_vm_mobile-opt-unittest-7-3600: skipping with 4/7 important changes since only 277/3600s have elapsed
2015-08-27 06:38:05-0700 [-] tests-fx-team-ubuntu64_vm_armv7_large-opt-unittest-7-3600: skipping with 4/7 important changes since only 47/3600s have elapsed

This same problem didn't occur the 26th, I looked at the logs.

I still don't understand why SETA skipping jobs stopped at 00:51.  I also don't understand why it started skipping jobs again at 6:37am.  Will investigate.

However, probably the machine pools were able to keep up with load overnight but when daytime ET hit, the number of pushes increased and caused the high pending counts w/o SETA.  

Looking at m-i there certainly was reason to skip jobs from midnight to 6am giving the intervals of pushes.
(Assignee)

Updated

3 years ago
Assignee: nobody → kmoir
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1199226
(Assignee)

Comment 2

3 years ago
Also, I wonder if there is something weird happening in bb because our masters have been up so long. They were due to be rebooted weeks ago but this has been deferred to the TCW Sat aug 29
URL: 1199552
(Assignee)

Comment 3

3 years ago
I suspect bug 1176784 is the cause.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1176784
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.