564084 - Implement signaling method between buildbot repack process and BYOB webapp

Status: RESOLVED WONTFIX

(Release Engineering :: Release Requests, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

As discussed in the BYOB meeting today, we need to figure out how the repack process on the buildbot slave is going to send notifications to the webapp.

The current plan is for repacks to be triggered by an hg commit, which means we can use a simple hg poller. However, we need to get status back to the webapp once the repack is done or fails out. Les says the current system works via a queue in the db, so it may be easiest just to have a final factory step set to alwaysRun that can report the outcome to the db regardless of failure status.

This bug we'll cover the actual work involved on the buildbot side. We'll hash out the details of how it's going to work in wiki page that Les will be creating under:

https://wiki.mozilla.org/Partnering:Projects:BYOB

Comment 1

[[e-mail] (:bhearsum)]

It may we worthwhile looking at writing a Status plugin that gets attached to the BYOB builders. alwaysRun steps may not complete in cases where the build is interrupted in a weird way.

Comment 2

[Les Orchard [:lorchard]]

I've tried jotting down a description of the workflow states a BYOB repack goes through, and how the current build queue works.  I'm open to revising / simplifying this stuff to get it working with buildbot.  Hopefully this helps:

https://wiki.mozilla.org/Partnering:Projects:BYOB:PilotBuildQueueDetails#Build_queue_tasks_in_BYOB

The main things I think that need extracting from this as an interface are:

* webapp->buildbot:
** perform repack build, move to private review hosting;
** move build to public release hosting;
** delete build files from private and public hosting

* buildbot->webapp:
** repack build success;
** repack build failure (plus when & why in human readable form)

The current build queue DB is overkill, so I'd be fine with reworking the webapp->buildbot part to whatever is easiest on the buildbot side. I'm not really familiar with buildbot, so let me know what works here (eg. drop a file in a directory somewhere? POST to a URL?)

For the buildbot->webapp side of things, it might be easiest to set up a URL on the webapp that buildbot can send a POST request to.  Then, I can handle whatever state changes & etc need to happen within the webapp data model.

Comment 3

[Les Orchard [:lorchard]]

(In reply to comment #2)
> The current build queue DB is overkill, so I'd be fine with reworking the
> webapp->buildbot part to whatever is easiest on the buildbot side. I'm not
> really familiar with buildbot, so let me know what works here (eg. drop a file
> in a directory somewhere? POST to a URL?)

Oh, I forgot that checking things into an hg repo was mentioned.  I could probably have the web app do this, but would need to hash out exactly what goes into the repo.

Currently, the "performBuild" queue task does a bit of work to get things ready for the repack python script:

http://viewvc.svn.mozilla.org/vc/projects/byob/trunk/application/hooks/builds.php?view=markup#l114

Since this stuff can be time-consuming, it might be nice if some or all of this stuff could be taken over by buildbot.  (eg. downloading / unpacking extensions, installing search add-ons, generating the distribution.ini, copying the builds to hosting, etc)

Comment 4

[Kev Needham [:kev]]

(In reply to comment #3)
> Oh, I forgot that checking things into an hg repo was mentioned.  I could
> probably have the web app do this, but would need to hash out exactly what goes
> into the repo.

Basically what we throw in the directories now. Coop provided the repack script we use with BYOB, so everything under distribution and extensions would go in, as well as the repack.cfg file. I'm hoping there's nothing we need to change other than writing the info permanently to a repo and committing it.

Comment 5

[Les Orchard [:lorchard]]

(In reply to comment #4)
> (In reply to comment #3)
> > Oh, I forgot that checking things into an hg repo was mentioned.  I could
> > probably have the web app do this, but would need to hash out exactly what goes
> > into the repo.
> 
> Basically what we throw in the directories now. Coop provided the repack script
> we use with BYOB, so everything under distribution and extensions would go in,
> as well as the repack.cfg file. I'm hoping there's nothing we need to change
> other than writing the info permanently to a repo and committing it.

Well, the potential issue is that it takes a bit to compose all the stuff that goes into the directories for a repack, before the repack script itself gets run.

Not minutes, usually, but often on the order of tens of seconds.  So, that's not something we want to do during the HTTP request when a user hits the button to request a build.

So, to do the prep work for a repack, either we still need the existing offline queue to run the PHP code or we need to move some form of that into buildbot-land on top of the repack python script.

This might start to sound kind of Rube-Goldberg-ian, but we could do something like this:

1. webapp checks a JSON file into hg
2. buildbot notices the updated JSON
3. buildbot runs a PHP script that generates the repack files from the JSON data
4. buildbot runs the Python repack script to perform the actual build

The PHP script in step 2 would basically be a tweaked version of what's run in my build queue right now, so the code changes could be minimal.

Someday, maybe in BYOB 2.0, we can consider switching everything over to Python like the rest of the cool kids doing webdev at Mozilla :)

Comment 6

[Chris Cooper [:coop] (he/him)]

I'm wading back into this finally. I think the comments here are missing some of the details I'll need to move forward though.

(In reply to comment #2)
> * webapp->buildbot:
> ** perform repack build, move to private review hosting;
> ** move build to public release hosting;
> ** delete build files from private and public hosting

The buildbot process can certainly create/move the repacks into a private review location. The review process (it's part of the web app, I'm guessing?) should handle the subsequent move/deletion.

> For the buildbot->webapp side of things, it might be easiest to set up a URL on
> the webapp that buildbot can send a POST request to.  Then, I can handle
> whatever state changes & etc need to happen within the webapp data model.

Agreed. If you want to get a syntax for PASS/FAIL setup, I'll write to that spec. I'll try to be verbose (within reason) about why things fail on the buildbot side, so make sure I can also pass a message string.

(In reply to comment #3)
> Since this stuff can be time-consuming, it might be nice if some or all of this
> stuff could be taken over by buildbot.  (eg. downloading / unpacking
> extensions, installing search add-ons, generating the distribution.ini, copying
> the builds to hosting, etc)

I think for the first-pass here we'll keep the process the same, i.e. the web app will continue to provide the repack.cfg + requisite dir structure. It might make sense eventually to have buildbot do that work, having all the required data to pull together the pieces contained in the repack.cfg.

IIRC, we've ruled out committing changes to hg as we do for partner repacks. Had we decided on an NFS fileshare as the alternative? I'll need to confirm that before I start writing a poller.

Following to that, will the repack config data be persistent on disk, or will it be re-created every time a new repack is required? This will affect how I consume new/changed config files and dirs from the builders.

How do you keep different repack.cfg files distinct in the web app-driven process? Are you currently dropping them each into their own dir based on some identifier, similar to the partner repacks? AIUI, you regenerate the repack.cfg files when a new repack is required, but if we're dropping the config files into a pending/ dir somewhere, they'll need some additional resolution in their filenames. Timestamp would likely suffice, or possibly identifier+timestamp.

Can we add key=value pairs to the repack.cfg files to facilitate BYOB repacking? Right now, we pass a bunch of command-line option to the repack script to know what version we're repacking, etc. In the cluster-of-slaves context, it makes more sense IMO to put all that info into the config file, and add an option to the repack script read directly from a single config file, e.g. partner-repack.py --config-file repack_partnerX_20110221172701.cfg

Comment 7

[Mike Taylor [:bear]]

Follow up from the meeting with Tomcat and Coop we just had about BYOB:

The repack data is now already generated and available.  What we need to do on RelEng is to identify repacks that need signing, sign them and move the signed bits back to BYOB space.

After talking it over we identified two use cases:
 1 - Sign a single repack
 2 - Sign all of the repacks for a new version of Firefox

Because the BYOB team will be generating the repack files the two use cases above could be handled by RelEng creating a service that monitors an incoming job directory for what I'm calling a job manifest.  This manifest really can be a path to the repack file and a path where to store the signed bits.

This service will then interact with the BYOB signing server to get the bits signed and to mark the job as complete by moving the job manifest to a job completed directory along with any log files that are generated.

bug 563798 is where I will track the setting up of the BYOB signing server.

Comment 8

[Chris Cooper [:coop] (he/him)]

Someone will pick these up in August.

Comment 9

[Chris Cooper [:coop] (he/him)]

http://kev.deadsquid.com/?p=1219