694616 - Verify docs.services.mozilla.com is updating on Github push

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Gregory Szorc [:gps]]

There were reports in IRC that the automatic hourly update/refresh of docs.services.mozilla.com on Mercurial push of services-docs isn't working.

Tarek said to file a bug so he can verify things when it isn't Friday night in France.

Comment 1

[Gregory Szorc [:gps]]

:atoll suggested in IRC that the docs updating might be a good Jenkins job. Something to consider...

Comment 2

[Tarek Ziadé (:tarek)]

it's up and running - closing

Comment 3

[Tarek Ziadé (:tarek)]

looks like we have the issue again

Comment 4

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

This fixed itself.

Comment 5

[Ryan Kelly [:rfkelly]]

So I'm not entirely clear on what's supposed to be happening here, but I pushed something to docs repo on github yesterday, and it hasn't appeared on the live website today.  Do we expect it to appear automatically?

Comment 6

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

afaik, docs doesn't pull from github. webops, keepers of this functionality, use svn for docs.s.m.c.

Comment 7

[Ryan Kelly [:rfkelly]]

OK, so we manually push from github to svn and then that goes out to the website automatically from there?  The title of this bug suggested that it should work automatically all the way through from github.

Comment 8

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

roping in webops on this. I honestly have no idea. :phrawzty can you assist?

Comment 9

[Daniel Maher [:phrawzty]]

Hello all,

First off, it may be helpful to provide some additional information about the environment in which we find ourselves.  I apologise if this digression is already known to you.

Generally speaking, we use internal source control from the admin nodes (which manage the webheads for a given cluster) to manage content.  In the past we used SVN for this purpose, though one some (newer) clusters, we use Git.  In either case, these repos are unrelated to Mozilla's general source control systems - it's really just for managing the content that gets assembled on the admin nodes and then pushed out to the wedheads.  In the case of the Static cluster[1] (which hosts docs.services.m.c, among others) we are using SVN.

In the specific case of docs.services.m.c, and unlike most of the other sites on the Static cluster, it would appear that the content on the admin node is managed from svn.mozilla.org[3] directly.  Looking at the contents of that repo, we can see that it contains a "compiled" site, which is to say HTML files instead of .rst sources.  Something, therefore, is acquiring the contents of the Github repo, processing it, and pushing it into svn.m.o.  I am currently engaged in tracking this mechanism down, and will update this bug once I have a clearer understanding of all of the moving parts.

[1] https://mana.mozilla.org/wiki/display/websites/Static+Cluster
[2] https://github.com/mozilla-services/docs
[3] http://svn.mozilla.org/projects/docs.services.mozilla.com/

Comment 10

[Richard Newman [:rnewman]]

(In reply to Daniel Maher [:phrawzty] from comment #9)
> Something, therefore, is acquiring the
> contents of the Github repo, processing it, and pushing it into svn.m.o.  I
> am currently engaged in tracking this mechanism down, and will update this
> bug once I have a clearer understanding of all of the moving parts.

Historically, that thing has been me. :P

Comment 11

[Daniel Maher [:phrawzty]]

Aaand I messed up the enumerated references in comment 9.  ¯\_(ツ)_/¯

Comment 12

[Ryan Kelly [:rfkelly]]

> Something, therefore, is acquiring the contents of the Github repo, processing it, and pushing it into svn.m.o.

I've done a couple of these pushes by hand in the past.  There's an update.sh script in the docs.services.mozilla.com svn that can be run to achieve this push, but I've no idea if or where or when it's run automatically.

Comment 13

[Daniel Maher [:phrawzty]]

Well, that would explain why it "stops working" sometimes - if the process is manual, then there's no guarantee that it will be triggered.  As far as I can tell neither update.sh nor update.py are run automatically from anywhere.

This would appear to be a good candidate for automation[1].  I propose that we simply add a cron job that pulls from the Github repo, builds the content, and then pushes it to the webheads.  Committing it to the Mozilla SVN is optional; however, if there are reasons for retaining this step (external tooling, perhaps?) then it can be retained as well.

Objections ?  General commentary ?

[1] http://imgur.com/KVun06L

Comment 14

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

+1 from me.

Comment 15

[Richard Newman [:rnewman]]

(In reply to Daniel Maher [:phrawzty] from comment #13)

> Objections ?  General commentary ?

+1 from me. I don't know of any reason why the SVN stage needs to continue to exist, but I acknowledge my limited frame of reference :P

Comment 16

[Ryan Kelly [:rfkelly]]

+1 automation ftw

Comment 17

[Daniel Maher [:phrawzty]]

As might be expected, the required packages for Sphinx support do not exist in any of the standard repositories.  I built packages for each dependency, and wouldn't you know, the chain requires for python-imaging and python-pillow, which contain conflicts (they both supply the PIL).

Next up: virtualenv.  On the plus side this means I can avoid distribution packaging concerns (much to the chagrin of OpSec); however, it comes with its own set of special problems - namely python 2.6, and the fact that subprocess.check_output isn't a thing, which means the supplied update.py will fail.

So instead there's Yet Another Bash Script (build_site.sh) which will clone the repo, install the necessary virtualenv components, and then build the site.  I've modified the cron job to reflect this change in the work flow; said cron job runs every 15 minutes.  Note that the Mozilla SVN repo is not part of this work flow.  Should this functionality be required, it can be added in to build_site.sh as necessary.

Comment 18

[Tarek Ziadé (:tarek)]

Thanks for making this happen.

Yeah, having a cron automatically running was the goal since day one. And at some point I ended up having a cron on my own server and it stopped working one day :)

For the record Pillow is a fork of Imaging. So if the first is installed you should be looking good

Comment 19

[Daniel Maher [:phrawzty]]

(In reply to Tarek Ziadé (:tarek) from comment #18)
> For the record Pillow is a fork of Imaging. So if the first is installed you
> should be looking good

Yes, it is - but RPM doesn't know that. :P  LOL dependency hell.

Comment 20

[Tarek Ziadé (:tarek)]

If the RPM package for python-pillow does not specify "Provides" and "Obsoletes" directives, that's a packaging bug

Comment 21

[James Bonacci [:jbonacci]]

So I guess we need a longer term plan for when this pipeline breaks...

from IRC on 4/7/2014:

Comment 22

[James Bonacci [:jbonacci]]

14:15 < alexis> ckolos: hey, do you know if the cron that updates docs.s.m.c is working okay?
14:17 < alexis> bobm: ^ ?
14:19 < ckolos> alexis: unknown.
14:20 < bobm> same here
14:20 < alexis> who should I ask about that?
14:21 < ckolos> https://bugzilla.mozilla.org/show_bug.cgi?id=964959
14:21 < ckolos> that's handled by the web ops team
14:22 < alexis> ah ok
14:22 < alexis> normally I'm happy to see that stuff is handled by phrawsty, but I'm not in this tz today.
14:22 < jbonacci> looking
14:23 < ckolos> there was another bug opened about it but my bz-fu is not working well
14:27 < ckolos> alexis: https://bugzilla.mozilla.org/show_bug.cgi?id=694616
14:27 < jbonacci> ckolos thanks
14:27 < tarek> yup
14:28 < alexis> ckolos: thx
14:29 < jbonacci> I touched both bugs to get them back into my queue in case this conversation comes up again
14:30 < tarek> I am still not sure what's the current status though
14:30 < tarek> e.g. is it syncing or not
14:31 < jbonacci> tarek yea I know
14:31 < jbonacci> we have two bugs but not process
14:31 < jbonacci> :-P
14:35 < tarek> OH: This docs automatic updates are talked about since 2011 :)
14:38 < jbonacci> zactly

Comment 23

[Daniel Maher [:phrawzty]]

I fixed this yesterday (in another bug that :alexis has visibility on).

In the interest of full disclosure, I no longer work in webops, so I'm removing myself from this bug.