394498 - Release automation should support for mirror-replication monitoring

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

As part of build process, we push bits out to mirrors, and then wait until "an acceptable amount" of mirrors have picked up the bits, before we proceed to the next steps in the build/release process.

In the past, "an acceptable amount" of mirrors was decided by waiting 8-12 hours.

However, with bug#390201, bouncer can now tell us what % of our total mirror bandwidth has picked up the newly pushed bits, which means we can now wait typically 4-6 hours, a big saving. Currently, this is through a webpage, so still requires human interaction.

bug#394494 tracks getting a command line util to do this same thing. Once this util is available, we should call this util from Bootstrap, so we can check mirror absorption as part of automation.

Comment 1

[J. Paul Reed [:preed]]

See my question in bug 394496, please; I'm confused about what Bootstrap is supposed to use this information for.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Once mirror absorption is enough to proceed, automation would send the notification email, telling QA its ok to proceed. This saves a human build engineer from manually polling bouncer, and then manually sending the same email.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaked bug dependency after closing bug#394494 as DUP.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Havent had time to work on this - putting back in the pool.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. Tweaking summary because at this point, we'd only do this for moz2-based releases, and they do not use bootstrap.

Comment 6

[Armen [:armenzg]]

I have been looking at r10249 from bug 408673 and it pretty much meet our needs to return a string indicating the uptake percentage for product and per o.s.

It seems that it requires CSE Python library (http://wiki.osuosl.org/display/Bouncer/cse+utils) according to bug 408673#c3 

I am thinking of where should this script be run to detect the proper percentage uptake. I will discuss this with Nick since he has given some thought in it (https://bugzilla.mozilla.org/show_bug.cgi?id=408673#c10)

John, we are looking into getting the average uptake of the 3 main platforms for a product and then shoot the email, right?

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> John, we are looking into getting the average uptake of the 3 main platforms
> for a product and then shoot the email, right?

I think we should look for a minimum number in 'available' for all product/platform combinations. Dave, how much availability do we normally want before we ship?

Comment 8

[Dave Miller [:justdave]]

I generally pick a number out of the air for each release based on traffic of previous releases and how much relative traffic we think this release will take.  In general for the past several releases, for a non-throttled security release which is auto-pushed to all users, I've been using 35000.  We opted for 45000 for the Firefox 3.5 release, but that was probably unwarranted (it had less traffic than a usual security release).

I so have Sentry pulling that number and putting it at the top of https://nagios.mozilla.org/sentry/ to make it easy to see, for the current releases.  The number shown there is the *lowest* number among the different types of downloads (direct, complete, partial) for all Windows platforms.  I don't even bother to look at Mac and Linux because those two are comparatively enough smaller in load that if we have enough to release Windows we have MORE than enough to release Mac and Linux... :)

Comment 9

[[e-mail] (:bhearsum)]

bug 508965 is tracking replacing Bouncer with MirrorBrain - not sure if we still want to spend time doing this with Bouncer.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> bug 508965 is tracking replacing Bouncer with MirrorBrain - not sure if we
> still want to spend time doing this with Bouncer.

That is, bug 482160.

Comment 11

[Dave Miller [:justdave]]

I would venture to say we're a month or two out from setting up MirrorBrain, at *least*.  Probably longer.  We need a web UI to manage it, and MirrorBrain doesn't have one at all right now, so our webdev is going to be writing that part.  The scope of what we wanted in the web UI isn't going to happen overnight, and we'll need some time to play around with it and test it before we outright switch.

Comment 12

[Armen [:armenzg]]

It is half of the quarter and I have not even touched this goal and the mobile-L10n infra still needs few more weeks. Let me know what to do with this.

Comment 13

[Armen [:armenzg]]

Havent had time to work on this - putting back in the pool.

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving to Future until we have time for this. :-(

Comment 15

[[e-mail] (:bhearsum)]

I'm planning to resolve this bug this quarter.

Comment 16

[[e-mail] (:bhearsum)]

The dependent bugs here are closed now, but we're still waiting for them to be rolled out before we can start work here. Frederic tells me this should be happening in the next couple of weeks.

Comment 17

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving over to Rail.

Comment 18

[Rail Aliiev [:rail]]

Attachment: Prototype

Attached is the first prototype.

It's implemented as change source for buildbot.

Comments are welcome.

Comment 19

[Rail Aliiev [:rail]]

Attachment: Implementation

Thanks for the comments on IRC.

Current version contains is available at: http://bitbucket.org/rail/tuxedo-client/src (attaching the current tip).

The has been split into 3 parts:

1. TuxedoClient library
2. tuxedo-cli (CLI script)
3. TuxedoPoller buildbot change source

Example usage is included in each part.

Comments are welcome.

Comment 20

[Rail Aliiev [:rail]]

Attachment: tuxedo-add

Updated version.

* Added --dry-run parameter to dump URLs and POST data without real HTTP requests.

* Added --platform parameter, so Solaris can be passed as well (multiple, default set to --platform win32 --platform linux --platform macosx)

* Path templates moved to the config file (--config). Example config embedded. Configure the file to get any FTP layout.

Comment 21

[Rail Aliiev [:rail]]

Attachment: tuxedo-add

Comment 22

[Rail Aliiev [:rail]]

Attachment: test-tuxedo-add

Attached is a simple bash script to test tuxedo-add script.

After submitting entries to Tuxedo (takes ~15 min here) it tries to download the list of the submitted files and check if the files exist (HTTP HEAD).

Staging Tuxedo account is required.

The scripts don't try to delete products, this should be done manually for safety.

Comment 23

[Rail Aliiev [:rail]]

Comment 20 and the following are not related to this bug (mess with bug 372746).

Going to reupload all attachments.

Comment 24

[Rail Aliiev [:rail]]

Attachment: lib part

Comment 25

[Rail Aliiev [:rail]]

Attachment: buildbotcustom/changes/tuxedopoller.py

Comment 26

[Rail Aliiev [:rail]]

Long time without any touch, back to pool.

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

What's left do do here?

Comment 28

[Rail Aliiev [:rail]]

(In reply to comment #27)
> What's left do do here?

IIRC,

1) Revise the code against current Tuxedo REST API
2) drop buildbot/bouncer platform mapping code, reuse buildbotcustom.common functions
3) add a builder which email the results to r-d@

Comment 29

[[e-mail] (:bhearsum)]

This is a P5 IMHO.

Comment 30

[Rail Aliiev [:rail]]

Ideas from catlee:

1) send email to r-d when uptake is enough for testing (10k)
2) send email to r-d when uptake is enough for release (45k?)
3) run backupsnip

Comment 31

[Chris AtLee [:catlee]]

(In reply to comment #30)
> Ideas from catlee:
> 
> 1) send email to r-d when uptake is enough for testing (10k)
> 2) send email to r-d when uptake is enough for release (45k?)
> 3) run backupsnip

Yeah, I'm thinking of something like a "push to mirrors" builder that:
- submits bouncer entries
- pushes to mirrors
- send email to r-d when uptake is enough for testing (10k)
- send email to r-d when uptake is enough for release (45k?)
- run backupsnip

Comment 32

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

Attached implements the approach of triggerable poller we talked on IRC:

* TuxedoUptakeClient simplified, we don't use uptake values per OS nor per installer, complete/partial mar. We use the minimum value of all products for all platforms.

* push to mirrors step (bug 540598) which already has revision, repo and release_config properties (required to know how to push to mirrors) triggers the scheduler and passes these properties.

* trigger() fires a deferred which gets release config from HG repo. After populating needed variables (version, oldVersion, productname, etc.), trigger() runs poll() in a loop. After a timeout (pollTimeout, 12 hours would be enough, IMO) polling will be stopped to prevent undead pollers.

* poll() gets the current uptake. If the uptake is good enough, it calls Triggerable.trigger to trigger the corresponding builders and stops the loop.

* In my test configuration I have 3 builders and 2 schedulers. 1st scheduler fires "ready_for_qa_builder" (which supposed to be a dummy builder with an email notification) and "final_verification" (already exists in our configuration). 2nd one fires dummy "ready_for_release_builder" builder.

c['schedulers'].append(TriggerBouncerCheck(
   name="ready_for_qa",
   builderNames=["ready_for_qa_builder", "final_verification"],
   min_uptake=qa_uptake,
   pollTimeout=30,
   username=None, password=None))

c['schedulers'].append(TriggerBouncerCheck(
   name="ready_for_release",
   builderNames=["ready_for_release_builder"],
   min_uptake=release_uptake,
   pollTimeout=60,
   username=None, password=None))

# Note copy_properties parameter

f1 = factory.BuildFactory()
f1.addStep(trigger.Trigger(schedulerNames=['ready_for_qa',
                                           'ready_for_release'],
                           copy_properties=['revision','repo',
                                            'release_config']))


Comments are welcome.

At the moment the following variables are semi-hardcoded (require reconfig):
addMARs, min_uptake, username, password. IMO, the only variable we may want to be version specific is min_uptake (may be lower for betas). If we want it to be version specific (and "reconfigless"), we need to add 2 variables to release_config.py (ready_for_qa_uptake, ready_for_release_uptake) and pass another variable to the new triggerable poller which tell us which variable to use for a particular instance. Not sure if it's worth to implement.

Comment 33

[Rail Aliiev [:rail]]

From our 1x1 with Chris:

* use getPage instead of urllib
* it would be great to have a generic triggerable poller class which accepts pollInterval and polling_fn parameters (at least)

Comment 34

[Chris AtLee [:catlee]]

Comment on attachment 491505 [details]
Triggerable poller

Nice approach.  As you mentioned already, using getPage and getting rid of threading is probably the right way to go here.

Also, not sure if lxml is available on all our masters.

Comment 35

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

(In reply to comment #33)
> * use getPage instead of urllib

Done.

> Also, not sure if lxml is available on all our masters.

xml.dom.minidom used instead.

Comment 36

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 491818 [details]
Triggerable poller

def get_release_uptake(tuxedoServerUrl, productName, version,
                       oldVersion=None, addMARs=True, username=None,
                       password=None):
    bouncerProduct = '%s-%s' % (productName, version)
    bouncerCompleteMARProduct = '%s-Complete' % bouncerProduct
    bouncerPartialMARProduct = '%s-Partial-%s' % (bouncerProduct, oldVersion)
    d = []
    d.append(TuxedoUptakeClient(tuxedoServerUrl=tuxedoServerUrl,
                                productName=bouncerProduct,
                                username=username,
                                password=password).getUptake())

In general, 'd' is a deferred, so I would prefer to see this called 'dl'.  Also, it looks like you're appending TuxedoUptakeClients, rather than deferreds that one of their methods return.  Can you set up a class method instead?  
  dl.append(TuxedoUptakeClient.getUptake(...))

As a further stylistic note, it looks like you've defined a class here merely as a container for several async functions.  It's much cleaner to define such a thing with nested methods:

def get_tuxedo_uptake(..):
  d = defer.succeed(None)
  def getUptakePage(_):
     ...
  d.addCallback(getUptakePage)
  def calculateUptake(page):
     ...
  d.addCallback(calculateUpake)
  return d

this has the advantage of nesting a sequence of operations into a function, as expected, and those operations execute more-or-less in lexical order.

You can write
    @d.addCallback
    def calculateUptake(page):
        ....
too, but that might be getting a bit fancy.

It *is* possible to write readable Twisted code.  You just need to plan ahead.

Comment 37

[[e-mail] (:bhearsum)]

Comment on attachment 491818 [details]
Triggerable poller

Where is this script going to live?

Please rename addMars to 'checkMars' or something similar.

There's a couple things here that should go in a library:
- bouncer product name generation
- bouncer url generation

Can you use the readReleaseConfig method from release.info?

Comment 38

[Rail Aliiev [:rail]]

(In reply to comment #37)
> Comment on attachment 491818 [details]
> Triggerable poller
> 
> Where is this script going to live?

It's not a script. It will be a part of buildbot-custom and will be running on a scheduler master.
 
There is a simple usage of it in comment 32.

I plan to put Tuxedo related functions somewhere to 'lib' and poller stuff to buildbot-custom/schedulers. I've used all-in-one-file approach just to simplify local testing.

> Please rename addMars to 'checkMars' or something similar.

I reused this name from tuxedo-add.py, but in the current context we can use checkMars, sure.

> There's a couple things here that should go in a library:
> - bouncer product name generation
> - bouncer url generation

Sure. Probably we can use tool's lib for this.

> Can you use the readReleaseConfig method from release.info?

It requires master side hg checkout, so we may have race conditions. The current implementation is very compact and works in memory.

I'll post an updated version tomorrow.

Thanks for the suggestions.

Comment 39

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

* Made on top of changes from  540598: push_to_mirrors_factory triggers mirror monitoring scheduler
* Minimum uptakes set to 10K and 45K
* Tuxedo functions moved to util/tuxedo.py

* Not sure where to put credentialsFromFile (and maybe rename it)
* Dummy builder requires an idle slave to run

I tested the patch in staging against an example xml file protected by .htaccess. It worked pretty smooth.

This patch and the patch from bug 588150 won't work without fixing external imports introduced in bug 588150.

Comment 40

[[e-mail] (:bhearsum)]

(In reply to comment #38)
> (In reply to comment #37)
> > Can you use the readReleaseConfig method from release.info?
> 
> It requires master side hg checkout, so we may have race conditions. The
> current implementation is very compact and works in memory.

Can you expand on this? I don't understand...

Comment 41

[[e-mail] (:bhearsum)]

Comment on attachment 493259 [details] [diff] [review]
Triggerable poller

I'd feel better about the tuxedo username and password being passed at config time. Since this is master side I don't think there's any reason we need to read it at runtime.

I think util.tuxedo belongs in build/tools/lib/python, not buildbotcustom. We already depend on tools for release masters, so that shouldn't be a big deal.

This is mostly there!

Comment 42

[Rail Aliiev [:rail]]

(In reply to comment #40)
> > It requires master side hg checkout, so we may have race conditions. The
> > current implementation is very compact and works in memory.
> 
> Can you expand on this? I don't understand...

Triggerable mirror monitoring code runs on masters and may have more than one instances. Unlike builders, schedulers (this one is a scheduler) have no unique work directory per instance. We're going to run more than one instances of this scheduler even if case if we support 1 branch: one for minimum (ready for qa) uptake and the second one for maximum (ready for release) uptake.

If we want to reuse the mentioned code and use local hg clones, we have to keep these clones in unique directories (a directory per instance) to prevent a mess with updating the files (hg up -r $TAG), which implies additional coding.

The current implementation is very simple and straightforward, IMHO.

Comment 43

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

(In reply to comment #41)
> I'd feel better about the tuxedo username and password being passed at config
> time. Since this is master side I don't think there's any reason we need to
> read it at runtime.

I used the following scenario. Is it OK?

import BuildSlaves # time to move to passwords.py?
......
mirror_scheduler1 = TriggerBouncerCheck(
......
    password=BuildSlaves.tuxedoPassword,
......


> I think util.tuxedo belongs in build/tools/lib/python, not buildbotcustom. We
> already depend on tools for release masters, so that shouldn't be a big deal.

Moved. See the next patch.
 
> This is mostly there!

Yay!

Comment 44

[Rail Aliiev [:rail]]

Attachment: Tuxedo part (tools/lib)

Comment 45

[[e-mail] (:bhearsum)]

(In reply to comment #42)
> (In reply to comment #40)
> > > It requires master side hg checkout, so we may have race conditions. The
> > > current implementation is very compact and works in memory.
> > 
> > Can you expand on this? I don't understand...
> 
> Triggerable mirror monitoring code runs on masters and may have more than one
> instances. Unlike builders, schedulers (this one is a scheduler) have no unique
> work directory per instance. We're going to run more than one instances of this
> scheduler even if case if we support 1 branch: one for minimum (ready for qa)
> uptake and the second one for maximum (ready for release) uptake.
> 
> If we want to reuse the mentioned code and use local hg clones, we have to keep
> these clones in unique directories (a directory per instance) to prevent a mess
> with updating the files (hg up -r $TAG), which implies additional coding.
> 
> The current implementation is very simple and straightforward, IMHO.

Ah, ok. I was proposing the alternative of reading the config from the master directory, but that has the disadvantage of not necessarily getting the tagged version. This is fine, then.

Comment 46

[[e-mail] (:bhearsum)]

(In reply to comment #43)
> Created attachment 493376 [details] [diff] [review]
> Triggerable poller
> 
> (In reply to comment #41)
> > I'd feel better about the tuxedo username and password being passed at config
> > time. Since this is master side I don't think there's any reason we need to
> > read it at runtime.
> 
> I used the following scenario. Is it OK?
> 
> import BuildSlaves # time to move to passwords.py?
> ......
> mirror_scheduler1 = TriggerBouncerCheck(
> ......
>     password=BuildSlaves.tuxedoPassword,
> ......

Yeah, that's fine.

Comment 47

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

Refreshed patch. Carrying r+.

Comment 48

[Rail Aliiev [:rail]]

Attachment: Triggerable poller

Added ready_for_qa, ready_for_release and final verification to notify_builders.

Comment 49

[Rail Aliiev [:rail]]

Comment on attachment 494347 [details] [diff] [review]
Triggerable poller

http://hg.mozilla.org/build/buildbotcustom/rev/b655d795a314

Comment 50

[Rail Aliiev [:rail]]

Comment on attachment 493377 [details] [diff] [review]
Tuxedo part (tools/lib)

http://hg.mozilla.org/build/tools/rev/561cceb9f7d1

Comment 51

[Rail Aliiev [:rail]]

Comment on attachment 494347 [details] [diff] [review]
Triggerable poller

I imported the older patch accidentally. :(
I backed out the commit and imported the proper one. 
http://hg.mozilla.org/build/buildbotcustom/rev/9c48a2f4a700

Comment 52

[Rail Aliiev [:rail]]

Attachment: BuildSlaves.py.template changes

Preproduction tests fail due to missing variables in BuildSlaves.py.template.

Comment 53

[Rail Aliiev [:rail]]

Comment on attachment 497527 [details] [diff] [review]
BuildSlaves.py.template changes

http://hg.mozilla.org/build/buildbot-configs/rev/838de9283db9

Comment 54

[Rail Aliiev [:rail]]

Comment on attachment 494347 [details] [diff] [review]
Triggerable poller

I've backed out this patch because it requires tools/lib/python in PYTHONPATH for every master, including those which don't use this scheduler.

As a possible solution I can move this scheduler into a separate file, so only masters used for releases (they have tools in PYTHONPATH already) use this scheduler. Other solution would be moving util.tuxedo from tools to buildbotcustom.

Comment 55

[Robert Kaiser]

(In reply to comment #54)
> I've backed out this patch because it requires tools/lib/python in PYTHONPATH
> for every master, including those which don't use this scheduler.

Hrm, now, after a backout of the backout, I run into exactly this problem on my master. Any way we can make things smoother, i.e. not failing with:

  File "/tools/buildbotcustom/buildbotcustom/scheduler.py", line 20, in <module>
    from util.tuxedo import get_release_uptake
ImportError: No module named util.tuxedo

?

Where does util.tuxedo live right now and why isn't it in buildbotcustom in the first place?

Comment 56

[Rail Aliiev [:rail]]

(In reply to comment #55)
> Where does util.tuxedo live right now and why isn't it in buildbotcustom in the
> first place?

Ooops, sorry for late reply.

util.tuxedo lives in tools repository. We've added tools/lib/python to PYTHONPATH environment for all master. Note that reconfig doesn't work if you change PYTHONPATH, you need a restart or a temporary symlinks of needed module directories to existing PYTHONATH (builbotcustom for example).

Comment 57

[Rail Aliiev [:rail]]

Bouncer API backend (Tuxedo) uses https for the REST API URLs, what requires
Python OpenSSL module installed on builder masters. IIRC, some of the masters
already has this module installed.

Bug 620641 filed

Comment 58

[Rail Aliiev [:rail]]

Attachment: Use "available" instead of "total"

A silly mistake. I should have used "available" instead of "total"...

Comment 59

[Rail Aliiev [:rail]]

Comment on attachment 503801 [details] [diff] [review]
Use "available" instead of "total"

http://hg.mozilla.org/build/tools/rev/34b89a7335a0

Comment 60

[Rail Aliiev [:rail]]

oldbugs--!

Comment 61

[Rail Aliiev [:rail]]

Verified during the Firefox 4.0b11 release cycle.