determine root cause of sudden surge in pending counts after enabling w10 talos

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
a year ago
5 months ago

People

(Reporter: kmoir, Assigned: alin.selagea)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

a year ago
Determine root cause why a large number of jobs were triggered for win10 and win7 when enabling win10 for talos in bug 1366029.  

10:38 AM 
<aselagea|buildduty> yeah.
10:39 AM 
<arr> CRITICAL Pending Jobs: 9290 on [t-w1064-ix]
10:39 AM 
<jmaher> we have 9000+ pending w10 jobs, odd
10:39 AM 
<arr> seems like that ramped up very fast
10:39 AM or we had a lot of old jobs in the database or... something
10:39 AM 
<jmaher> that seems suspicious
10:40 AM 
<aselagea|buildduty> no, all of them are waiting for <30 minutes in the queue
10:41 AM 
<arr> who just triggered 10K tests?
10:42 AM 
<jmaher> I suspect catlee's theory
10:43 AM 
<arr> jmaher: that the puppet patch didn't land?
10:44 AM 
<jmaher> arr: yes, it did yesterday
10:44 AM 
<arr> jmaher: which theory?
10:44 AM 
<jmaher> arr: old pending jobs are scheduled

side note: this may be similar to the issue in bug 1223042
(Reporter)

Updated

a year ago
Assignee: nobody → aselagea
(Reporter)

Comment 1

a year ago
Created attachment 8873159 [details]
Screen Shot 2017-05-31 at 3.31.35 PM.png

high pending counts from this morning
(Reporter)

Updated

a year ago
Summary: determine root cause of sudden surge on high pending counts due while enabling w10 talos → determine root cause of sudden surge in pending counts after enabling w10 talos
(Reporter)

Comment 2

a year ago
Alin, did we find out any information on what the root cause might be here for the post-mortem?
Flags: needinfo?(aselagea)
(Assignee)

Comment 3

a year ago
Sadly, no. 

Something similar happened in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1223042, but we couldn't find an answer in that case either.
Flags: needinfo?(aselagea)
(Reporter)

Comment 4

a year ago
I think too much time has passed to effectively determine the root cause.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX

Updated

5 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.