1006918 - automated push to docs.services.mozilla.com doesn't seem to be updating

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ryan Kelly [:rfkelly]]

(This may be a return of the issues in Bug 964959, so I'm cc'ing everyone from that bug).

IIRC anything committed to the "production" tag at https://svn.mozilla.org/projects/docs.services.mozilla.com should be deployed onto https://docs.services.mozila.com, but the current version does not seem to be deployed.

The current rev in svn as I see it is:

"""
$ svn info
Path: .
Working Copy Root Path: /home/rfk/repos/mozilla/services/docs.services.mozilla.com
URL: https://[e-mail]/projects/docs.services.mozilla.com
Repository Root: https://[e-mail]
Repository UUID: 4eb1ac78-321c-0410-a911-ec516a8615a5
Revision: 128017
Node Kind: directory
Schedule: normal
Last Changed Author: [e-mail]
Last Changed Rev: 127961
Last Changed Date: 2014-05-06 15:48:51 +1000 (Tue, 06 May 2014)
"""

But its contents do not match what's live on the site.

Comment 1

[Jake Maul [:jakem]]

You are correct! It seems to be broken. Here's what it tries to do:

[[e-mail]1 docs.services.mozilla.com]# ./build_site.sh
/data/static/src/docs.services.mozilla.com/_git /data/static/src/docs.services.mozilla.com
Already up-to-date.
/data/static/src/docs.services.mozilla.com
/data/static/src/docs.services.mozilla.com/_git /data/static/src/docs.services.mozilla.com
Running Sphinx v1.2.1

Exception occurred:
  File "conf.py", line 4, in <module>
ImportError: No module named mozilla_sphinx_theme
The full traceback has been saved in /tmp/sphinx-err-5KYtof.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-users/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
/data/static/src/docs.services.mozilla.com



It appears the build_site.sh script is attempting to verify that some Sphinx packages are installed, then using them to build the actual site. Note this it pulls everything from git, *not* from SVN:

https://github.com/mozilla-services/docs


Interestingly, the build_site.sh script itself is neither in SVN nor in git. It's rather simple:

#!/bin/bash

if [ ! -d "_git" ]; then
    git clone https://github.com/mozilla-services/docs _git
else
    pushd _git
    git pull
    popd
fi

if [ ! -d "_git/lib/python2.6/site-packages/sphinx" ]; then
    pushd _git
    virtualenv --no-site-packages .
    bin/pip install Sphinx
    bin/pip install sphinxcontrib-seqdiag
    popd
fi

pushd _git
bin/sphinx-build -b html -d build/doctrees source ../
popd


Any theories?

Comment 2

[Ryan Kelly [:rfkelly]]

Ah, this is new then, I thought we were still deploying from a built version in svn.

The repo actually has its own build logic, something like this might be a bit more robust in the face of changing dependencies:


#!/bin/bash

if [ ! -d "_git" ]; then
    git clone https://github.com/mozilla-services/docs _git
else
    pushd _git
    git pull
    popd
fi

pushd _git
make ./venv/bin/sphinx-build
./venv/bin/sphinx-build -b html -d build/doctrees source ../
popd

Comment 3

[Ryan Kelly [:rfkelly]]

And to be clear, the "docs.services.mozilla.com" svn repo is no longer involved in this process at all?  I'll be quite glad to delete it from my local system.

Comment 4

[Toby Elliott [:telliott]]

The only svn I'm aware of is svn+ssh://svn.mozilla.org/projects/services.mozilla.com/trunk

That one autodeploys to https://services.mozilla.com/

Comment 5

[Tarek Ziadé (:tarek)]

another option is to just publish to readthedocs and get a CNAME for docs.services.mozilla.com there and stop struggling with this process where we don't know who is in charge... :)

Comment 6

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

+1 on tarek's suggestion.

Comment 7

[James Bonacci [:jbonacci]]

+1000 on that suggestion. I have been hearing about this since 2011

Comment 8

[Tarek Ziadé (:tarek)]

step #1 is to have our own little server to proxy calls to readthedocs, see https://docs.readthedocs.org/en/latest/alternate_domains.html#cname-ssl

So, Chris, can that be a dev- AWS small box in your opinion ? If so - I am happy to set it up and ping you for changing the CNAME location

Comment 9

[Jake Maul [:jakem]]

readthedocs sounds fine to me, but that's outside my purview so until that's accepted I'll keep working on fixing the current system.

(In reply to Ryan Kelly [:rfkelly] from comment #2)
> Ah, this is new then, I thought we were still deploying from a built version
> in svn.
> 
> The repo actually has its own build logic, something like this might be a
> bit more robust in the face of changing dependencies:
> 
> 
> #!/bin/bash
> 
> if [ ! -d "_git" ]; then
>     git clone https://github.com/mozilla-services/docs _git
> else
>     pushd _git
>     git pull
>     popd
> fi
> 
> pushd _git
> make ./venv/bin/sphinx-build
> ./venv/bin/sphinx-build -b html -d build/doctrees source ../
> popd

This 'make' ultimately yields:

ImportError: enchant C library not found

Is this looking for the "Enchant" spell checking library, or something else? Spell checking doesn't make much sense to me since this is ultimately a static site... no idea what it would be checking. Is this a new dependency?

Comment 10

[Tarek Ziadé (:tarek)]

Frankly, this process of pushing to a svn is not robust because it relies on a complex building. readthedocs handles this fine imho, and that's less maintenance for us.

Jake, what do you need to consider this solution accepted ?

Comment 11

[Jake Maul [:jakem]]

Ignore my last... it does indeed appear to be the Enchant spellcheck library... don't know why it's needed, but installing it makes it work. This is now updated... thanks for the improved deploy script.

I see it's changed to a sort of sandstone theme. Nice. :)

This improved deploy mechanism does not use the SVN repo at all... only git. If the SVN repo was used only as an intermediary for this, then you can safely remove it entirely.

I'm going to close this out since the original request is now fulfilled. I do recommend carrying on the readthedocs discussion somewhere, though... that may well be a better solution. I don't have any problem with using that instead, it's just something that'd normally be self-serviced by the project/product owners and not managed by IT, so I wouldn't be the one to set it up or make that sort of decision. FWIW, several other Mozilla webdev projects use it and seem to be happy with the results.

Comment 12

[Tarek Ziadé (:tarek)]

oh well - let's have the readthedoc discussion if we hit the issue again ;)  Thanks Jake!