679390 - configure scl1 caching servers to cache addon-sdk tarballs

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Myk Melez [:myk] [@mykmelez]]

To enable test slaves to download tarballs of the Add-on SDK without requiring us to mirror the contents of the product's git repository in hg, the caching servers in scl1 should be configured to permit a gitmirror post-commit hook to push tarballs to those servers that they then cache and make available to test slaves.

Comment 1

[Brian Warner [:warner :bwarner]]

I can start writing the post-commit hook (bug 679395) as soon as you let me know how this is going to look. Potential forms it could take:

 * tarballs are created on gitmirror, in the post-commit hook, left in a public_html/ directory there
 * tarballs are created on gitmirror then scp'ed off to some other host for distribution
 * post-commit hook triggers a script on the distribution machine, which does a 'git pull' then builds tarballs

I also want to find out if I should be building a script to update some sort of pointer file (like a "current-version.txt" that contains the filename of the latest tarball), so the buildbot scripts can reliably fetch the right tarball. Also, we need to figure out how the buildbot gets triggered: it needs to be *after* the tarball is ready to download, which might mean that the post-commit hook should do the notifying itself, after doing the tarball stuff.

I don't need the server work to be done before writing the hook, but once I know the plan I can pipeline the work a bit.

Comment 2

[Corey Shields [:cshields]]

Need to meet with noahm and oremj on this, and right now our focus has been on getting AMO in phx1 on Thursday (fingers crossed).  Expect us to sync up after that.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

Corey: what's the status of this?  Brian is ready to write the necessary code as soon as y'all decide how you want the tarballs delivered to the caching servers.


(In reply to Brian Warner [:warner :bwarner] from comment #1)
> I also want to find out if I should be building a script to update some sort
> of pointer file (like a "current-version.txt" that contains the filename of
> the latest tarball), so the buildbot scripts can reliably fetch the right
> tarball. Also, we need to figure out how the buildbot gets triggered: it
> needs to be *after* the tarball is ready to download, which might mean that
> the post-commit hook should do the notifying itself, after doing the tarball
> stuff.

These sound like RelEng questions.

Lukas: any thoughts on these?

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

> (In reply to Brian Warner [:warner :bwarner] from comment #1)
> > I also want to find out if I should be building a script to update some sort
> > of pointer file (like a "current-version.txt" that contains the filename of
> > the latest tarball), so the buildbot scripts can reliably fetch the right
> > tarball. Also, we need to figure out how the buildbot gets triggered: it
> > needs to be *after* the tarball is ready to download, which might mean that
> > the post-commit hook should do the notifying itself, after doing the tarball
> > stuff.
> 
> These sound like RelEng questions.
> 
> Lukas: any thoughts on these?

Talked it over with Dustin and it sounds like we could put a poller on the git-mirror which in order to determine what the tarball would be called, then have a build step that polls the ftp dir until that tarball name appears to trigger the rest of the steps for that change.

Comment 5

[Corey Shields [:cshields]]

I'm blocking on this - assigning it out to someone who has more time to implement.

Brian, hit me up on IRC for some background.

Comment 6

[Brian Hourigan [:digi]]

Hi Bryan/Myk

I will be working with you to get this setup. Are the tarballs currently on gitmirror anywhere? I can periodically pull them from git onto hg1/hg2 and allow you to download them via HTTP.

Comment 7

[Brian Warner [:warner :bwarner]]

Hm, I don't know. I'm not very clear on the role of 'gitmirror'.. who owns that?

I just met with joduinn to explain what we're looking for from the jetpack side. The workflow we discussed would be cleanest with a git.mozilla.org (bug 528360) that can serve gitweb-style tarballs to buildslaves. Once that's in place, the buildbot needs to be configured to pull from it (somehow). So I think he's got some thinking to do about the buildbot, which might drive this bug a bit further.

Comment 8

[Corey Shields [:cshields]]

(In reply to Brian Warner [:warner :bwarner] from comment #7)
> Hm, I don't know. I'm not very clear on the role of 'gitmirror'.. who owns
> that?
> 
> I just met with joduinn to explain what we're looking for from the jetpack
> side. The workflow we discussed would be cleanest with a git.mozilla.org
> (bug 528360) that can serve gitweb-style tarballs to buildslaves. Once
> that's in place, the buildbot needs to be configured to pull from it
> (somehow). So I think he's got some thinking to do about the buildbot, which
> might drive this bug a bit further.

When we spoke about this a month or so ago getting the tarballs to the build network via the mirrors is a stop gap to having git.mozilla.org, because unfortunately we don't have the time or resources to turn up another VCS right now (and do it right).

We had talked about pulling your github repo in to gitmirror[1], tarring it up, and sending that tarball to the hg mirrors in the build network  (not via hg - that just happens to be their name and purpose.  these tarballs would serve up over http)

That was the whole point of this bug.  If this has changed we should talk and re-sync, but getting git up and going as a service is still preempted by a lot of things.

[1] - https://mana.mozilla.org/wiki/display/websites/gitmirror.mozilla.org

Comment 9

[Corey Shields [:cshields]]

sorry for the shuffle, this goes back to server ops queue

Comment 10

[Myk Melez [:myk] [@mykmelez]]

Brian and Cory: it's great to finally see forward progress on this!  So I really don't want to say this, but...

After chatting with John O'Duinn and Brian Warner about it today, I think we should put this on hold temporarily, as we have identified a possible way to move forward on the work that we have been blocked on (and until which this stopgap measure is intended to tide us over).

So let's just pause this for a week or so while John pursues that avenue.  Then, if it turns out that he won't be able to move forward on that work, we can resume our efforts here.

Comment 11

[Brian Hourigan [:digi]]

Myk: Is anything needed here or can we close this bug? If you ever want to resume this we can look at it again in a new context.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

Let's keep it open for a few more days to give John time to report back on whether that alternative avenue for moving forward is feasible, since if it isn't, we'll want to move forward with the workaround that includes a fix for this bug.

John: any update on your investigation?

Comment 13

[Myk Melez [:myk] [@mykmelez]]

I talked to John, and he's making good progress on a better alternative to the workaround we identified in this bug, so I will close this one in the expectation that John will succeed in implementing the alternative.  If it doesn't end up happening, we can always reopen this bug then.