520274 - Turn off queue compression in production pool-o-talos-slaves

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P4)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Original talos had dedicated slaves per build. This meant that some slaves would get backlogged on work, while others were idle. On some busier branches, the slaves would never be able to catch up. To make sure the backlog cleared, queue compression was turned on from day one. This means that when a slave became free, if there was a backlogged queue, the slave would jump to do the last job, and throw away all the other pending jobs.

Pooling the slaves addressed this.

Out of paranoia, we left queue compression turned on when we moved to pool of slaves. However, as best as we can tell, the pooled slaves are well able to keep up. We should now turn this off and see if this holds true.

Comment 1

[Chris AtLee [:catlee]]

Attachment: Turn off merging on talos pool

Turn off merging of requests in talos pool for everything except mozilla-1.9.0 builds.

Also, I've left merging enabled in the talos staging pool, otherwise we'd fall hopelessly behind and start failing because old builds have been deleted.

Comment 2

[Chris AtLee [:catlee]]

Deferring to the Future until after the minipocalypse.

Comment 3

[alice nodelman [:alice] [:anode]]

Comment on attachment 404626 [details] [diff] [review]
Turn off merging on talos pool

Removing review request.  Will require a new, non-bitrotted patch when we decide to do this.

Comment 4

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 5

[Chris AtLee [:catlee]]

This was always the intent, but I don't think we'll ever be willing to do this.

WONTFIX?

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> This was always the intent, but I don't think we'll ever be willing to do this.
> 
> WONTFIX?

Lets see how testpool load looks after lsblakk lands her developer-config-file to chose what jobs are run in TryServer. I expect that to have a big reduction in load to Tryserver.

Comment 7

[Armen [:armenzg]]

Attachment: Turn off queue compression for all talos jobs on all branches

How is bug 588908 different than this one?
I don't think I understand properly the difference.

catlee how does the patch look?

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> Created attachment 469875 [details] [diff] [review]
> Turn off queue compression for all talos jobs on all branches
> 
> How is bug 588908 different than this one?
> I don't think I understand properly the difference.

That bug is specific to the try branch.  This bug is for ALL branches.

> catlee how does the patch look?

The patch looks good in general.  Did you mean to get rid of the 'enable_opt_unittests' settings?

Also, I'm surprised that all branches are running the same talos test suites.  Have you checked that nothing except the merging flag has been changed?

Comment 9

[Armen [:armenzg]]

I would like to WONTFIX this bug as I believe we would not be able to handle the load on the minis.

We could file a new one when we have enough minis.

What do you say?

Comment 10

[Chris AtLee [:catlee]]

Agreed.