807351 - Move Mozillians Update Script To /bin/update_site.py

Status: VERIFIED FIXED

(Participation Infrastructure :: Phonebook, defect, P1)

Description

[Ben (:bensternthal)]

Currently the update script for mozillians is maintained on the admin node for the cluster, it is not in a public repository.

We would like to move this into /bin/update_site.py within the project and IT's shell script will just call this.

Comment 1

[Brandon Burton [:solarce]]

Here are the parts of the shell script that could be moved into the python script 

CODE_DIR="/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians"
VENDOR_DIR="$CODE_DIR/vendor"
LOCALE_DIR="$CODE_DIR/locale"

	echo -e "Updating code..."
	cd $CODE_DIR
	git fetch origin -q
	checkretval
	
	git checkout -f origin/master -q
	
	git submodule sync -q
	git submodule update --init --recursive -q
	checkretval

	echo -e "Updating vendor..."
	cd $VENDOR_DIR
	git submodule sync -q
	git submodule update --init --recursive -q
	checkretval

	cd $LOCALE_DIR
	find . -name "*.mo" -delete
	svn up
	checkretval

	./compile.sh
	checkretval
	
	cd $CODE_DIR
	find . -type f -name '.gitignore' -or -name '*.pyc' -delete
	checkretval

    cd $CODE_DIR
    python -W ignore ./manage.py collectstatic --noinput
    checkretval

	cd $CODE_DIR
	python -W ignore ./manage.py compress_assets
	checkretval
	
	cd $CODE_DIR
#   python vendor/src/schematic/schematic migrations
    python -W ignore manage.py syncdb
	checkretval
    python -W ignore manage.py migrate
	checkretval

	cd $CODE_DIR
    python -W ignore manage.py update_product_details -f 2>&1
	checkretval

	cd $CODE_DIR
    git rev-parse HEAD > $CODE_DIR/media/revision

	cd $CODE_DIR
	python -W ignore ./manage.py cron index_all_profiles
	checkretval

checkretval is a function that confirms each command returns a 0

function checkretval()
{
    retval=$?
        if [[ $retval -gt 0 ]]
        then
                $error "Error!!! Exit status of the last command was $retval"
                exit $retval
        fi
}

This is used by the shell script to abort if anything returns 0, then cron sends an error email

Let us know when you have a version to test on dev

Comment 2

[Jake Maul [:jakem]]

erm... confusion here, Mozillians is deployed by Chief now, and there's a different (Python) script that handles it.

The simple solution is to just check in the update.py script that Chief uses into your git repo. AMO and SUMO do it this way:

https://github.com/mozilla/kitsune/tree/master/scripts/update

Note that *their* script is not *your* script... that's just an example.

There's a separate commander_settings.py as well, which functions a bit like a settings_local.py file and would not be in git. You can see a sample .dist file in there.


I can send a pull request that gets you started if you like.

Comment 3

[Giorgos Logiotatidis [:giorgos]]

:solarce thnx for the script

:jakem please help us with a help request!

Comment 4

[Jake Maul [:jakem]]

https://github.com/mozilla/mozillians/pull/335

Comment 5

[Jake Maul [:jakem]]

Once that's merged we'll need to update the config for the Chief app to call your script instead of our local copy. We'll also need to put the commander_settings.py config file in place.

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/mozillians

https://github.com/mozilla/mozillians/commit/c492a6d0b5e1d16a6d7fd3c1e9160b44a8d9e235
[bug 807351] Moving Chief/Commander update scripts into repo.

https://github.com/mozilla/mozillians/commit/a09e25fb31b5806a86930476910c2b4905d41aae
[bug 807351] Add ES index update command.

Comment 7

[Giorgos Logiotatidis [:giorgos]]

@jakem I merged your pull request and added a function to re-index ES upon merge. I call this function in update().

We have this code now in dev. When we push it to stage we need to create commander_settings.py, right?

Comment 8

[Giorgos Logiotatidis [:giorgos]]

Jakem:

The update.py script is now on stage. Can you please activate it?

Thanks!

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/mozillians

https://github.com/mozilla/mozillians/commit/07a968e104d101831f106f680034c0392f2e00d8
[bug 807351] Add collectstatic command.

Comment 10

[Ben (:bensternthal)]

Jake can you activate this on stage. 

Pushing new code to stage (and having it pass tests) is dependent on this.

Comment 11

[Matt Brandt [:mbrandt]]

How soon can we get this activated on stage? QA is seeing our test suite fail (which appears linked to this bug) - https://bugzilla.mozilla.org/show_bug.cgi?id=816264#c3 - we've xfailed the failing test for the time being.

Comment 12

[Ben (:bensternthal)]

Jake, do I need to move this to a server ops bug?

Comment 13

[Jake Maul [:jakem]]

This is completed. The commander_settings.py is moved to the proper (new) place, and the Chief config is updated to point to your copy of the update.py script (at scripts/update/update.py).

Leaving open, pending a green light to do the same on prod.

Comment 14

[Giorgos Logiotatidis [:giorgos]]

Tried to push to stage and got this error:

"""
Traceback (most recent call last):
  File "/usr/bin/commander", line 4, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/commander/main.py", line 101, in main
    cmd_mod = import_cmdfile(cmdfile)
  File "/usr/lib/python2.6/site-packages/commander/main.py", line 82, in import_cmdfile
    return imp.load_source('cmdfile', cmdfile)
IOError: [Errno 2] No such file or directory
"""

Comment 15

[Ben (:bensternthal)]

Jake can you help us debug the error Giorgos received.

Comment 16

[Ben (:bensternthal)]

Bumping up the priority. We are now blocked from pushing live.

Comment 17

[Brandon Burton [:solarce]]

I am looking into

Comment 18

[Jake Maul [:jakem]]

This is fixed on stage now. Prod is not converted to Chief yet, so if you're in need of a push there rather quickly, let's do that the old-fashioned way and not block on converting it while under duress.

Comment 19

[Ben (:bensternthal)]

Ok thanks, we need to wait for giorgos to double check everything is working properly.

We will not push live/convert under duress.

Comment 20

[Giorgos Logiotatidis [:giorgos]]

Works as promised on stage, thnx :solarce!

Comment 21

[Ben (:bensternthal)]

Jake:

Do i need to file a new bug to get this task complete on production or should I re-open this one.

Comment 22

[Giorgos Logiotatidis [:giorgos]]

(In reply to Jake Maul [:jakem] from comment #18)
> Prod is not converted to Chief yet,

Jake, I'm not sure I understand your comment. We are using Chief on prod already. Can we please activate the same script on production?

Thanks!

Comment 23

[Jake Maul [:jakem]]

Sorry, that was poorly worded. I meant to say, prod was not converted over to use *your* update script, and was still using ours.

This has just been completed. Prod is now using scripts/update/update.py, just like stage does.

I haven't run a deploy this way, but I don't expect any trouble. If you have any trouble with it, please let me know.

Comment 24

[Matt Brandt [:mbrandt]]

Gently nudging to QA verified