419981 - Talos-related builds need priority treatment getting through the virus-scanning system on stage

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task, P2)

Description

[Dave Miller [:justdave]]

The new virus scanning on stage-new takes about 45 minutes to scan the contents of a push from the l10n build system (because it's about 2.2 GB of compressed stuff that has to be opened and looked at).  Since, with the new system, stuff doesn't even show up on the ftp server until after it's been scanned, this means if something destined for a Talos box gets pushed at the same time as a bunch of l10n stuff, it will take until that virus scan completes before it gets pushed.  We need to find some way to scan those in parallel or give them priority treatment to avoid things getting backed up.

Comment 1

[Dave Miller [:justdave]]

Nick: can I get one of the following from you?

Either 1) A list of the stuff that needs priority treatment, or 2) a list of the stuff that should have deferred treatment.

Comment 2

[Dave Miller [:justdave]]

Whichever one can be described with the most-specific subdirectory is what I need.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The current set of dirs that need a fast path is 
  
 firefox/tinderbox-builds/
    FX-WIN32-TBOX-trunk/
    bm-xserve08-trunk/
    fx-linux-tbox-trunk/
    production-prometheus-vm-mozilla1.8/ 
    prometheus-vm-mozilla1.8/      *
    prod-pacifica-vm-mozilla1.8/
    pacifica-vm-mozilla1.8/        *
    bm-xserve05-mozilla1.8/
    bm-xserve02-mozilla1.8/        *

The starred directories won't be required after bug 417147 is fixed.

If l10n turns out to a blocker on other builds (like nightlies), we will probably want to invert this and make a slow-path.

Are builds pushed live immediately after being scanned, or does a group all get pushed at once ?

Comment 4

[Dave Miller [:justdave]]

anything that's there when the scan starts gets pushed all at once after the scan finishes.

Basically all I plan to do is segregate them so we have two rsync and scan processes running in parallel.  One that specifically grabs the fast-track stuff and one that grabs everything else.  The nothing will delay the fast track except other stuff in the fast track.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The transition for bug 417147 happened yesterday, so please ignore the starred dirs.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Any progress to report ?

Dunno why this isn't marked blocking the Q1 improvements, it has been in our discussions.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Updated list of fast-path dirs:
 /pub/mozilla.org/firefox/tinderbox-builds/
    mozilla-central-linux/
    mozilla-central-macosx/
    mozilla-central-win32/
    actionmonkey-linux/
    actionmonkey-macosx/
    actionmonkey-win32/
    FX-WIN32-TBOX-mozilla1.9.0/
    bm-xserve08-mozilla1.9.0/
    fx-linux-tbox-mozilla1.9.0/
    production-prometheus-vm-mozilla1.8/ 
    prod-pacifica-vm-mozilla1.8/
    bm-xserve05-mozilla1.8/

Extracted from the talos config here:
 http://mxr.mozilla.org/seamonkey/source/tools/buildbot-configs/testing/talos/perfmaster/master.cfg

Comment 8

[Jeremy Orem [:oremj]]

Changing QA Contact.